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(57) Abstract 

A system for monitoring standard broadcast signals, comprises a device for receiving the broadcast signals, a user-operable selection 
device for inputting criteria identifying program content of interest to the user, a database for storing data representing the criteria, and a 
recognition device for generating from the broadcast signals a program data stream representative of the program content A comparator 
compares the program data with the stored data, and an output device carries out a predetermined action, such recording a segment of the 
program, when the program data matches the stored data. Tbe system will, for example, activate a window on a multimedia PC when 
preselected program material is present in the broadcast signals. 
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AUTOMATIC BROADCCAST MONITORING SYSTEM 

This invention relates to a system for automatically 
monitoring broadcasts, such as television broadcasts, and 
detecting content of particular interest to individual 
viewers . 

Many organizations, for example, stock brokerage 
services, have an on-going need to monitor world events. It 
is known for such organizations to retain media and news 
scanning services that employ people to watch television to 
look for particular topics of interest. These services log 
occurrences of particular items and either advise their 
customers by telephone or provide reports that may include 
video clips that they manually put together. For example, 
the service might continuously record a particular 
television channel while someone watches it. The person 
watching will record in a log the occurrences of interest, 
noting the time and tape position. The recorded tapes will 
later be searched using the time or tape position from the 
logs. Edited clips are then put together in a time consuming 
manual process. 

The shortcomings with this method are many. The 
response time in noticing an event of interest is slow, 
typically minutes. Many hours may pass before the 
interested party is informed of the event. The cost of 
paying people to watch television is high. For example, at 
$10 per hour, twelve hours per day, the cost works out at 
$43,800 per channel per year. Looking up individual video 
clips is very slow and requires knowledgeable staff because 
the look-up is keyed to time of occurrence rather than words 
or phrases. Typical tape fast forwards cr rewinds take 
three to five minutes each. The approach is also inflexible 
in that any new search on recorded video requires people to 
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watch it all over again, thus incurring the entire overhead 
again. 

Other solutions include raw television capture devices 
for computers. These capture devices convert the television 
video to digital format for storage and distribution on 
computer networks. These products capture the raw footage, 
but are not in any way dependent on content. They may 
capture and distribute video, and/or audio, and/or caption 
text, but they do not alert users to particular topics 
immediately as they occur on television broadcasts. 

Another solution uses a technique called datacasting. 
With this technique, the broadcaster injects data into the 
vertical blanking interval (VBI) of the video television a 
signal that contains information about the program. The data 
is created by the broadcaster, or source program author. The 
data usually contains titles, summaries, and other 
information related to the program. This information can be 
used by the user to select items of interest. The 
disadvantage to this approach is that it requires special 
information to be created at the source and transmitted from 
the source. It also requires special proprietary hardware to 
receive the data, and more significantly still requires the 
user to continually monitor the datacast information. This 
VBI data injection approach has another major shortcoming in 
that very few, if any broadcast stations use this technique, 
and those that do encode it so that special equipment is 
required to decode it. CNN at Work, sold by Intel is an 
example of such a product. 

An object of the invention is to alleviate the afore- 
mentioned disadvantage of the prior art. 

SUMMARY OF THE INVENTION 
According to the present invention there is provided a 
system for monitoring broadcast signals, comprising: means 
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for receiving the broadcast signals and generating program 
signals to make available programs carried by said broadcast 
signals; user-operable selection means for inputting 
criteria identifying program content of interest to a user; 
a database for storing data representing said criteria; 
recognition means for generating from said broadcast signals 
a program data stream separate from said program signals and 
representative of program content; a comparator for 
comparing said program data stream with said stored data; 
and an output device operable to carry out a predetermined 
action when said program data stream matches said stored 
data to permit the user to examine a portion of a monitored 
program associated with said matching program data stream. 

The broadcast signals can be any type of signals 
carrying programs and distributed over a suitable 
transmission medium, wireless, cable or satellite. In one 
embodiment, the broadcast signals are television signals 
(although radio signals can in some cases be used if they 
are accompanied by program data, as is sometimes the case) . 
The may also be television signals distributed digitally, 
for example, using the MPEG format. 

In the case of television signals, the recognition 
means can monitor the closed caption text normally 
transmitted with the signals. Upon detection of certain 
words or phrases, the system generates an alarm condition to 
alert the user to the presence of such program content. For 
instance, in the case of a standard television set, a simple 
system could just turn up the volume so as to allow the user 
to hear the audio. In a more sophisticated system, the 
signals are stored and tagged for subsequent retrieval using 
database searching techniques. 

The predetermined action can be any form of action that 
permits the user to examine the portion of the program of 
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interest. For example, it can involve recording the program 
signals and data signals for later review, activating a 
video window in a computer, or just activating a beeper to 
alert the user to the broadcast of program content of 
interest. For example, a message could be sent to a beeper 
advising the user that -Clinton is on television now", thus 
enabling him to watch the program on the nearest television 



set 



In another embodiment voice or pattern recognition 
techniques can be employed to monitor the audio or video 
directly. 

Thus, with the system according to the invention, users 
can be automatically alerted to events as they happen. 
Video, audio and closed captions from television broadcasts 
can be automatically indexed and stored so they can be 
retrieved later in whole or in part, using ad hoc queries. 
Users can retrieve individual clips of video, audio or 
closed caption text by specifying content, for example by 
searching for a key word or phrase in addition to the 
traditional time/position based queries. 

The invention is thus capable of automatic detection of 
content in television broadcasts using computers, 
synchronizing and capturing the incoming closed captioned 
text together with the video and audio in a computer 
environment, and indexing and retrieval of individual 
sections of video and audio based on the content of, for 
example, closed caption text. 

The invention is applicable to the field of multimedia 
applications generally. It provides techniques for 
automatically creating indexed multimedia databases and has 
many applications. For example, government elected 
representatives and their assistants can monitor television 
programs to determine what is being said about them, about 
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their policies, etc. They can be alerted to issues 
immediately. 

Regulated or government organizations can use this 
invention to monitor television programs for negative (or 
positive) views on their organizations. They can know about 
problems reported in the media and react sooner. They can 
track the media reaction to their policies. 

Marketing departments can use this invention to collect 
information on competitors, customers of the organizations 
that employ them. They can be alerted to issues immediately. 
As well, marketing multimedia databases can be built 
automatically. 

Broadcast industry, cable carriers and telephone 
companies can use this system to create and maintain 
multimedia databases. Cable Companies and Telco's can use 
this invention to provide differentiated services to their 
customers, for example filtering and querying capability on 
live broadcasts and also on multimedia databases. 

Individuals can use this technology to minimize time 
spent watching television. Items not of interest can be 
filtered out. Program transcripts can be captured and 
perused with only the desired video and audio being played 
back. 

The invention also provides other multimedia 
applications and databases with access to fully indexed 
video, audio and closed captions from broadcast television. 

The invention also provides a multimedia computer 
system, comprising: a central processing unit; a video 
capture unit for receiving broadcast television signals and 
generating a video signal therefrom; a video display unit 
capable of displaying said video signals under program 
control; means for decoding data signals embedded in said 
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television signals and generating a program data stream 
representative of program content of said television 
signals; means for inputting criteria identifying program 
content of interest to a user; and a database for storing 
data representing said criteria. The central processing unit 
is programmed to continually compare the program data with 
said stored data and carry out a predetermined action when 
said program data matches said stored data to permit the 
user to examine a portion of a monitored program associated 
with said matching program data stream. 

The invention still further provides a method of 
monitoring broadcast signals, comprising the steps of: 
extracting program signals from said broadcast signals; 
generating from the broadcast signals a program data stream 
separate from said program signals and representative of 
program content; comparing said program data stream with 
stored data identifying program content of interest to the 
user; and generating an output condition when said generated 
program data stream matches said stored data to permit the 
user to examine a portion of a monitored program associated 
with said matching program data stream. 

The invention additionally provides a computer-readable 
storage medium having binary information encoded thereon, 
said binary information being operable when interpreted by a 
computer to direct the computer to monitor broadcast signals 
and generate program signals to permit the presentation of 
program material carried by the broadcast signals to a user, 
analyze incoming broadcast signals to generate a program 
data stream separate from said program signals and 
representative of their program content, compare said data 
stream with stored data identifying program content of 
interest, and generate an output condition when said 
generated data matches said stored data to permit the user 
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to examine a portion of a monitored program associated with 
said matching program data stream. 

BRIEF DESCRIPTION OF THE DRAWINGS 
The invention will now be described in more detail, by 
way of example only, with reference to the accompanying 
drawings, in which: - 

Figure 1 is a block diagram of a system in accordance 
with the invention showing the overall system architecture; 

Figure 2 is a block diagram showing the internal 
architecture of the system shown in Figure 1; 

Figure 3 is a flow chart showing the closed caption 
stream processing; 

Figure 4 is a flow chart showing audio stream 
processing; 

Figure 5 is a flow chart showing video stream 
processing; 

Figure 6 is a flow chart showing the alert handling 
procedure; and 

Figure 7 is a flow chart showing the stored data 
processing; 

Figure 8 is an embodiment of the invention based on a 
standard stand-alone multimedia personal computer; 

Figure 9 is a functional block diagram of a live feed 
arrangement for the embodiment shown in Figure 8; 

Figure 10 is a functional block diagram of a network 
feed arrangement; 

Figure 11 is a functional block diagram of an 
arrangement for playing back or re-recording a video 
segment; 

Figure 12 shows a typical display screen; 
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Figure 13 shows a typical display screen showing a 
program listing; and 

Figure 14 is a block diagram of another embodiment of 
the invention applicable to an on-demand architecture . 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

In Figure 1, a monitoring system 1 in accordance with 
the invention receives a standard broadcast television 
signal from a tuner 2. The system is connected over a bi- 
directional link 6 to a multimedia display 3 consisting of a 
personal computer including a CRT monitor 4 and speakers 5. 
The multimedia display can form part of a conventional 
multimedia PC set-up. The system 1 can also be connected to 
a network 7, which in turn is connected to individual 
workstations 8. 

The monitoring system 1 comprises a video capture unit 
9, a closed caption capture unit 10, and an audio capture 
unit 11. Such units are per se commercially available. Data 
streams 14, 15, 16, which are preferably digital although 
they could be analogue, from these units are passed to bus 
13 of unit 12, shown in more detail in Figure 2. Unit 12 
carries out several functions as follows: 

• Video, audio and closed Caption data streams are 
synchronized and tagged for time of arrival. 

• Users are alerted to program content by continuous 
checking of the closed caption text stream. 

• Video, audio and closed captions are saved to disk 
storage. The video and audio clips can be retrieved 
by word or phrase using the closed captioned text 
and the synchronizing tags that index the video and 
audio. The Video, audio and closed caption text 
might be saved to disk continuously but the 
recording of the data can also be based on content. 
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For example, the system might be instructed to start 
recording video, audio and closed caption data from 
30 seconds before (using a delayed feed buffer) to 5 
minutes after the appearance of a keyword in the 
closed caption stream. This is significant since 
continuous storage of video data imposes huge 
storage requirements. 

• Clients on the network 7 can receive alerts, 

multimedia feeds and perform queries as shown on the 
multimedia display above. 

In operation, the user enters data either through, for 
example, a LAN-connected PC 8 or the workstation 3. For 
example, the user might enter a series of key words 
representing topics of interest. These key words are then 
entered into a profile database in mass storage device 20 
along with the identity of the user. When the selected key 
words appear in the closed caption data stream, the system 
generates an alert signal to alert the user. For example, 
the system might be set to trigger an alert if the words 
CLINTON and YELTSIN occur within an adjacent stream of 
twenty words. In a simple case, the alert may, for example, 
involve turning up the sound on the monitor 4. if desired, 
the system can also employ fuzzy logic principles to find 
similar words, for example, synonyms of the words entered by 
the user. This arrangement also copes with mis-spellings, 
which are quite common in closed caption systems. 

Each user connected to the system can of course enter 
his or her own customized search profile. 

Referring now to Figure 2, when the system is 
monitoring the broadcast signals, the incoming data streams 
14, 15, 16 are passed over internal bus 13 to comparator 17, 
which compares the data with comparison data stored in a 
profile database 18 set-up by the user in mass storage unit 
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20. Database 18 stores the data entered by the user through 
the multimedia PC 3. For example, the user may have chosen 
a set of keywords and phrases to look for in the closed 
caption data stream. These are entered through multimedia PC 
3 and stored in the database 18. 

The incoming data streams, or as noted above selected 
portions of them, are stored in mass storage device 20 along 
with a time stamp added by time stamp unit 21. Storage 
device 18 may be analogue, although it is preferably digital 
to allow random access to the information. 

Through PC 3, the user can also set up other actions 
for the system to perform when a keyword or phrase is found. 
These actions include sounding a beep, activating a remote 
beeper, opening a video window, sending electronic mail - 
essentially any action the computer can be programmed to 
perform. Also the system can automatically create a profile, 
(or comparator database), from an existing database, 
spreadsheet or word processor file, that can be used for 
monitoring. The system creates this profile by searching the 
specified files for proper nouns, categories and pre-defined 
keywords. After retrieving these from the specified file, 
they are presented to the user for modification. 

Each user 8 on the network can also add, delete, and 
modify the database profile 18 over the network 7 to 
customize it for his individual requirements. 

When system is storing data, the closed captioned text 
is stored in a standard text database in the mass storage 
unit 20. There are at least two columns in the database, 
namely the captured phrase and the absolute time it was 
captured. The audio stream and video streams are also 
captured together with their absolute start time. Individual 
sections of video and audio can be retrieved using SQL-like 
queries on the closed captioned text. The associated video 
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and audio are retrieved using absolute time returned from 
the text query. The system will position to the exact time 
within the video and audio that the word or phrase occurs 
at. 

The system also has a delayed feed buffer 19, 
implemented as a circular buffer. This stores that last n 
minutes of each of the data streams. The user can specify 
the length of this buffer and each of the three data streams 
can have different lengths. This buffer 19 also provides the 
user with the ability to backup a few seconds or minutes 
when an alert is received. Using technology known per se, 
the delayed feed buffer can also be played back at double 
speed so that the user can catch up to the live broadcast. 

The system provides the ability to distribute video, 
audio and text in whole or in part. Lower speed systems may 
choose only to receive the closed captioned text of a 
program, or just the audio. 

A specific example of the use of the monitoring system 
is as follows: A television viewer wants to watch the movie 
"Gone With The Wind". The viewer sets the computer to beep 
when the words "I don't give a damn" are spoken. 
Alternatively, having stored the same movie with this 
technology, they could position directly to the scene where 
those words are spoken, without having to know where in the 
movie, by time or relative position, these words occur. The 
user interface for positioning to the words or phrases of 
interest is very similar to a word or phrase search in a 
word processor, or an SQL query. 

The system in the embodiment described monitors the 
incoming closed captioned text stream, synchronizes it with 
the incoming video and audio, and alerts the user to topics 
of interest. When an alert happens, programmable actions, 
such as sounding a beep and/or bringing up a video window, 
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occur. The incoming video, audio and closed caption text 
are stored with time based tags and indexes, creating a 
fully indexed multimedia database automatically. 

Users have an SQL-like query capability on closed 
caption text with an ability to retrieve associated portions 
of video and audio. They can configure the monitoring 
software for particular words, phrases or general content, 
and perform particular actions upon alerts. 

The video, audio and closed captioned text can be 
distributed on the computer network 7, either directly or 
through electronic mail. The system and network managers 
can be provided with an ability to manage the system and 
network resource usage. 

Figure 3 shows the processing of the embedded data 
stream, which in this embodiment constitutes the closed 
caption stream. The text is first acquired in unit 110 and 
after insertion of a time-stamp in unit 111 inserted into 
the delayed feed buffer 19, after which is compared to 
comparator 113 for comparison with words from the user 
profile stored in storage unit 20. If a match is found, 
decision unit 114 activates alert handling unit 115 (shown 
in more detail in Figure 6) . 

In the absence of a match, decision unit 116 determines 
whether the closed caption text is to be recorded in 
accordance with user instructions. If yes, the closed 
caption text is written to file in unit 117; if not, unit 
118 determined whether the text is to be displayed in closed 
caption display 119. 

Figure 4 is a flow chart similar to Figure 3 for 
pattern matching based on audio, which is treated separately 
from the video. Audio units 210 to 219 operate in a similar 
manner to units 110 to 119 in Figure 3 except for the fact 
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that the pattern matching is achieved by extracting text 
from the audio stream using voice recognition techniques. 

Figure 5 is a flow chart similar to Figure 3 for 
pattern matching based on video, which is treated separately 
from the audeo. Video units 310 to 319 operate in a similar 
manner to units 110 to 119 in Figure 3 except for the fact 
that the pattern matching is achieved by extracting text 
from the video stream using pattern recognition techniques 
or by analyzing color histograms, for example. 

Figure 6 shows how alerts are handled. When an alert 
occurs as a result of a pattern match, unit 410 identifies 
the alert type. If an audible alert has been requested, 
decision unit 411 sounds bell 412. 

Decision unit 413 determines whether a recording has 
been requested, and if so causes unit 414 to write video, 
audio, and closed caption data from delayed feed buffer 19 
to be written to a file on mass storage unit 20. Unit 415 
causes the recording of incoming video, audio, and closed 
caption data to be commenced. 

Decision unit 416 determines whether a display has been 
requested, and if so unit 417 causes unit 418 to start 
displaying the video, audio and closed caption data. 

Figure 7 illustrates the stored data handling. One or 
more keywords are first requested from the user by unit 510. 
The user enters the keywords through the computer keyboard. 

Unit 511 searches the closed caption file for matching 
keywords and unit 512 displays a list of matching segments. 
Unit 513 requests a user selection. Decision unit 514 
returns control to the request unit 510 if no selection is 
made; otherwise unit 515 locates the matching segment, which 
is then displayed by unit 516. 
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A system in accordance with the invention can 
conveniently be implemented as a stand-alone system on a 
personal computer running a Windows™ operating system as 
shown in Figure 8. Personal computer 30 includes at least a 
486 DX66 and preferably a 90MHz Pentium or higher 
microprocessor 31 on the motherboard and a hard drive 32 
with preferably at least one gigabyte of storage space 
connected to PC bus 35. A Creative Labs Video Blaster™ 
video capture card 33 is located in one of the expansion 
slots of the motherboard. A suitable 16-bit sound card 34 , 
such as a Soundblaster™, is located in a second expansion 
slot. The personal computer 30 is connected to a 
conventional video display unit 28 and keyboard 29. 

The video card 33 and sound card 34 send respective 
compressed video packets 35 and audio packets 36 to the PC 
bus 35. 

Incoming rf television signals 37 from a cable or 
antenna source 38 are fed to a cable converter or VCR which 
demodulates the incoming rf signals and generates an audio 
stream 40, which is fed to the sound card 34, and a video 
stream 41, which is fed to the video card 33. 

The video signals 41 are also fed to a Sunbelt 
TextGrabber™ closed caption decoder 42. This decodes the 
closed caption signals embedded in the vertical blanking 
interval and sends an ASCII text stream to the PC bus 35 
through the serial port 44 and UART 45, which created text 
packets 46. 

The user enters instructions through the keyboard 29 
with the assistance of dialog boxes and a mouse (not shown) 
in a customary manner. 

Figure 9 shows in more detail the processing of the 
signals in the personal computer 30. The video signals are 
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passed through rate controller 50, which measures the frame 
rate in accordance with user instructions, to video delay 
unit 51. The audio signals are passed directly from audio 
capture unit 34 to audio delay unit 52. The closed caption 
signals are passed through closed caption pattern matching 
unit 53, which looks for pattern matches with the stored 
data representing the selected criteria, and closed caption 
display unit 54 to closed caption delay unit 55. Caption 
pattern unit 53 can carry out pattern matches on the basis 
of Boolean combinations of keywords entered by the user 
through the keyboard 29. For example, the pattern matching 
circuit 53 might trigger an event if two selected words 
occur within a stream of twenty adjacent words. The closed 
caption display unit 54 permits the user to view the closed 
caption text directly in a window on the video display unit 
28 under program control. 

The delay units 51, 52, 55 serve as buffers that hold 
the respective incoming signals for a time that can be 
selected by the user, for example 10 seconds or 20 seconds. 
This allows the user to either replay a portion of program 
material that has just been broadcast or to arrange for the 
computer to store not only a predefined segment of program 
material after detection of a pattern match, but also a 
segment prior to finding a match. For example, a typical 
set-up may be to store the previous ten seconds and the 
following twenty seconds after a pattern match. Each delay 
unit 51, 52, 55 has three output ports a, b, c. 

The video delay unit 51, audio delay unit 52, and 
closed caption delay unit 55 are connected through port b to 
a network interface 56 for the distribution of captured 
material over a network, such as a local area network. 

The above units are also each connected through port c 
to a synchronizing unit 57, an audio synchronizing unit 58 
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and a closed caption synchronizing unit 59. These three 
units 57, 58, 59 create an AVI (Audio-Video interleaved) 
file 60 on hard disk 32, which contains a video clip 
consisting of video, audio and associated closed caption 
data. 

The units 51, 52, and 55 are also connected through 
port a to additional synchronizing units 61, 62, 63 
connected respectively to a video display generator 64, a 
sound generator 65 in audio card 34, and a closed caption 
display generator 66. The video and closed captioning thus 
appear in a window, with the video (and audio) synchronized 
to the closed captioning. 

When a pattern match is detected by the unit 53, a 
signal 68 is sent to the record callback unit 67. This 
causes one or more of several predetermined actions to take 
place depending on the mode selected by the user. The record 
callback unit activates ports a, b or c of delay units 51, 
52, 55 to cause the delayed feed to be sent to the live 
display through units 61, 62, 63, the network 56, or the AVI 
file 60 through units 57, 58, 59. Instead of being stored in 
a computer file, the video and audio (and possibly closed 
caption stream) could be stored on a VCR tape. The ports can 
access the buffered signals at any time point in the buffers 
depending on the user setting. 

Figure 10 shows an arrangement where the computer 
receives a feed from a local area network (IAN) . In this 
case the incoming broadcast signals are received by a 
network server, digitized and transmitted to the clients on 
the network. The network provides signals equivalent to the 
capture units 33, 34, 42. The rest of the system remains the 
same. In this embodiment, only the server requires a video 
card and closed caption capture unit. The clients receive 
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the digitized signals, including video, sound, and closed 
captions over a network interface, such as SMC EtherEZ. 

In Figure 11, an AVI file is received by unit 90 and 
the audio extracted in unit 95. The video is sent through 
video synchronizer 91, rate monitor 92 and video splitter 93 
to video display 94 and output AVI file 97. The closed 
caption stream is sent through closed caption synchronizer 
100, closed caption sink 99, which extracts closed caption 
text for continuous display, closed caption splitter 98, 
from where it is passed to closed caption display 96 and AVI 
output file 97. The units 94 and 96 allow the video to be 
displayed along with its associated closed caption text. The 
output file 97 allows the user to edit and re-record 
portions of the stored text. 

Figure 12 shows a typical display screen running in an 
active window 70 on visual display unit 28. Video segment 71 
is playing along with its associated audio. Beside the video 
segment 71 is the closed caption text 72. Pointer 73 
indicates the portion of text related to the current image 
71. The pointer 73 can be dragged up and down with the mouse 
to cause the video corresponding to the indicated text to be 
played. 

Figure 13 shows a window displaying a listing of 
recorded video clips. Double clicking on any of the items 
listed causes the item to appear in an active window along 
with the associated text for playback, which can be 
initiated using a Windows™ style control bar. 

The system can be programmed to watch multiple channels 
simultaneously. In this case each channel has its own 
processing system as described with reference to Figure 9. 

In the embodiment of Figure 14, block 700 contains a 
monitoring system as described above and outputs digital 
signals for distribution over a remote network. This is 
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achieved by feeding the signals through satellite gateway 
703 to satellite antenna 704, which sends them via satellite 
705 to any one of a series of clients, such as 706. The 
clients 706 are connected back to the block 700 via the 
internet 707 or the public switched telephone network 708. 
The client 706 can use a service known as DirecPC, which 
provides a wide bandwidth downlink, for example, to the 
internet, with the uplink being provided through a 
conventional modem. 

Unit 702 receives broadcast signals 701, for example 
from air or over a cable. These can be conventional analogue 
signals, or alternatively they can be digital signals, for 
example in MPEG format. The unit 702 receives instructions 
from client 706 over the network. When a pattern match is 
achieved, it send the program signals over the network to 
the client 706. 

The system described in Figure 14 effectively acts as a 
network, with the unit 700 being the network server. 

In a further enhancement, the client 706 is connected 
back to a media server 709, which permits access to "on- 
demand" channels. A second monitoring system 710 can be 
provided, if desired, both units 706 and 710 being 
controllable by the clients 706. 

The described system thus has the advantage that it 
offers instant alert to events as they occur. Television 
feed can be automatically organized as it is stored; i.e., 
closed caption text is synchronized with the incoming video 
and audio. This provides the ability to automatically 
create a multimedia database from which items can be 
retrieved based on content, such as spoken words, and the 
ability to retrieve video and audio segments with word or 
phrase resolution. A stored video feed can be retrieved 
based on new information 
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The synchronized video, audio and closed captions can 
be distributed over computer networks. 

The invention also offers a much lower cost due to the 
elimination of manual intervention, and a reduction of raw 
information flow, resulting in less network congestion. 
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We claim: 

1. A system for monitoring broadcast signals, comprising: 

a) means for receiving the broadcast signals and 
generating program signals to make available programs 
carried by said broadcast signals; 

b) user-operable selection means for inputting criteria 
identifying program content of interest to a user; 

c) a database for storing data representing said 
criteria; 

d) recognition means for generating from said broadcast 
signals a program data stream separate from said program 
signals and representative of program content; 

e) a comparator for comparing said program data stream 
with said stored data; and 

f) an output device operable to carry out a 
predetermined action when said program data stream matches 
said stored data to permit the user to examine a portion of 
a monitored program associated with said matching program 
data stream. 

2. A system as claimed in claim 1, wherein said 
recognition means decodes data signals embedded in said 
broadcast signals to produce said program data stream. 

3. A system as claimed in claim 2, wherein said embedded 
data signals comprise a closed caption text stream. 

4. A system as claimed in claim 1, wherein said 
recognition means comprise audio recognition means for 
analyzing audio content of the broadcast signals. 

5. A system as claimed in claim 1, wherein said 
recognition means comprise pattern recognition means for 
analyzing video content of the broadcast signals. 

6. A system as claimed in any of claims 1 to 3, further 
comprising a mass storage device for storing said program 
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signals in response to a match of said program data stream 
and said stored data. 

7. A system as claimed in claim 6, wherein said program 
signals are stored with their associated program data on 
said mass storage device. 

8. A system as claimed in claim 6, wherein said program 
signals are indexed according to said program data. 

9. A system as claimed in claim 6, further comprising 
video and audio capture devices for generating respective 
video and audio signals providing said program signals. 

10. A system as claimed in claim 6, further comprising 
means for tagging the stored program signals with the time 
of broadcast. 

11. A system as claimed in claim 1, wherein said output 
device generates an alert signal to alert the user to the 
presence of program content matching the user's selection 
criteria. 

12. A system as claimed in claim 11, wherein said alert 
signal activates a video monitor. 

13. A system as claimed in claim 12, wherein said video 
monitor is a television set. 

14. A system as claimed in claim 12, wherein said video 
monitor forms part of a multimedia computer environment. 

15. A system as claimed in claim 14, wherein said 
multimedia computer environment also provides said user- 
operable selection means. 

16. A system as claimed in claim 6, further comprising a 
short-term buffer for storing said program signals for a 
predetermined period to permit access to program content for 
a preselected time prior to detection of a program content 
match . 
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17. A system as claimed in claim 1, which is connected to a 
computer network to permit clients thereon to individually 
perform queries and receive program signals from the system 
according to their respective program criteria. 

18. A system as claimed in claim 11, wherein said output 
device transmits said alert signal to a remote location for 
reception by the user. 

19. A system as claimed in claim 18, wherein said alert 
signal is a beeper signal. 

20. A system as claimed in claim 1, wherein said broadcast 
signals include a plurality of channels, said recognition 
means generate a plurality of data streams corresponding to 
said respective channels, and said comparator continually 
compares said program data contained in said plurality of 
program data streams with said stored data so as to monitor 
simultaneously said plurality of channels. 

21. A system as claimed in claim 1, wherein said database 
stores a plurality of predefined user profiles representing 
preselected criteria for respective users, and means are 
provided to activate any one of said predefined user 
profiles. 

22. A system as claimed in claim 1, wherein said broadcast 
signals are television signals, and said recognition means 
generates said program data stream from signals transmitted 
in the vertical blanking interval between successive 
television frames. 

23. A system as claimed in claim 1, wherein said broadcast 
signals are radio signals. 

24. A system as claimed in claim 1, which is located in a 
network server, said network server sending said program 
signals over the network to clients located thereon in 
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response to a match between said program data stream and 
said stored data. 

25. A system as claimed in claim 24, which is located in a 
network server, wherein said network server forms part of a 
local area network. 

26. A system as claimed in claim 24, wherein said network 
server is located remotely from said clients and a long- 
distance communications link is established between said 
network server and said clients. 

27. A multimedia computer system, comprising: 

a) a central processing unit; 

b) a video capture unit for receiving broadcast 
television signals and generating a video signal therefrom; 

c) a video display unit capable of displaying said 
video signals under program control; 

d) means for decoding data signals embedded in said 
television signals and generating a program data stream 
representative of program content of said television 
signals; 

e) means for inputting criteria identifying program 
content of interest to a user; 

f) a database for storing data representing said 
criteria; and 

said central processing unit being programmed to 
continually compare the program data with said stored data 
and carry out a predetermined action when said program data 
matches said stored data to permit the user to examine a 
portion of a monitored program associated with said matching 
program data stream. 

28. A multimedia computer system as claimed in claim 27, 
wherein said data signals are embedded in the vertical 
blanking interval of said television signals. 
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29. A multimedia computer system as claimed in claim 28, 
wherein data signals are closed caption signals. 

30. A multimedia computer system as claimed in claim 29, 
wherein said stored data represent one or more keywords 
selected by the user. 

31. A multimedia computer system as claimed in claim 30, 
wherein said stored data represent a plurality of keywords, 
and said output condition is generated when said plurality 
of keywords are detected in said program data stream within 
a preselected period in accordance with predefined Boolean 
logic. 

32. A multimedia computer system as claimed in claim 27, 
wherein said central processing unit is programmed to store 
a segment of said video signals in response to a match of 
said program data and said stored data, said segment being 
stored with associated program data. 

33. A multimedia computer system as claimed in claim 32, 
wherein said segment has a length preset by the user. 

34. A multimedia computer system as claimed in claim 33, 
further comprising a buffer for temporarily storing said 
video signals prior to comparison of the associated program 
data with said stored data, said computer being programmed 
to store a segment of said video signals for a period 
starting from a preselected amount of time prior to 
detection of a match of said program data and said stored 
data and finishing a preselected amount of time after a 
match of said program data and said stored data. 

35. A multimedia computer system as claimed in claim 34, 
wherein said central processing unit stores multiple said 
segments indexed according to said program data. 

36. A multimedia computer system as claimed in claim 27, 
wherein said computer system has an operating system 
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employing multiple windows, and said video signals appear in 
one or more of said multiple windows under program control. 

37. A multimedia computer system as claimed in claim 36, 
wherein said video signals appear in one or more of said 
multiple windows in response to a match of said program data 
and said stored data. 

38. A multimedia computer system as claimed in claim 36, 
wherein said central processing unit is programmed to 
display a listing of stored segments in one of said windows 
in response to a user command. 

39. A multimedia computer system as claimed in claim 36, 
wherein text information derived from said program data 
stream is displayed in one of said windows. 

40. A multimedia computer system as claimed in claim 39, 
wherein said displayed text information is synchronized with 
associated an associated video display. 

41. A multimedia computer system as claimed in claim 40, 
wherein said computer is programmed to indicate text related 
to a displayed image. 

42. A multimedia computer system as claimed in claim 41, 
wherein said computer is programmed to display video 
corresponding to text information pointed to by the user so 
that when the user points to a string of text the computer 
displays the video image associated with the text. 

43. A multimedia computer system as claimed in claim 27, 
which forms part of a network server, said network server 
distributing program signals to clients on the network in 
response to said output condition or a user reguest. 

44. A multimedia computer system as claimed in claim 43, 
wherein said network is a local area network. 
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45. A multimedia computer system as claimed in claim 43, 
wherein said clients are connected to said network server 
over long-distance communications links. 

46. A method of monitoring broadcast signals, comprising 
the steps of: 

a) extracting program signals from said broadcast 
signals; 

b) generating from the broadcast signals a program data 
stream separate from said program signals and representative 
of program content; 

c) comparing said program data stream with stored data 
identifying program content of interest to the user; and 

d) generating an output condition when said generated 
program data stream matches said stored data to permit the 
user to examine a portion of a monitored program associated 
with said matching program data stream. 

47. A method as claimed in claim 46, wherein said program 
data signals are extracted from signals embedded in said 
broadcast signals. 

48. A method as claimed in claim 46, wherein said broadcast 
signals are television signals and said data signals are 
embedded in the vertical blanking interval thereof. 

49. A method as claimed in claim 48, wherein said embedded 
data signals are closed caption signals. 

50. A method as claimed in claim 4 6, wherein said output 
condition generates an alarm signal to alert the user to the 
presence of matching data. 

51. A method as claimed in claim 4 6, wherein said output 
condition activates a video monitor to permit the user to 
watch the program material. 

52. A method as claimed in claim 4 6, wherein said output 
condition causes program material contained in said 
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broadcast signals to be stored so as to permit the user to 
watch the program material at a later time. 

53. A method as claimed in claim 52, wherein said program 
material is stored with its associated program data. 

54. A method as claimed in claim 53, wherein said video 
monitor forms part of a multimedia computer system. 

55. A method as claimed in claim 46, wherein said broadcast 
signals are received in digital format. 

56. A method as claimed in claim 46, which takes place in a 
network server, and where said program signals are sent to 
clients on the network. 

57. A method as claimed in claim 47, wherein said program 
signals are transmitted over a long-distance communications 
link to a remote said client in response to said output 
condition or a client request. 

58. A computer-readable storage medium having binary 
information encoded thereon, said binary information being 
operable when interpreted by a computer to direct the 
computer to monitor broadcast signals and generate program 
signals to permit the presentation of program material 
carried by the broadcast signals to a user, analyze incoming 
broadcast signals to generate a program data stream separate 
from said program signals and representative of their 
program content, compare said data stream with stored data 
identifying program content of interest, and generate an 
output condition when said generated data matches said 
stored data to permit the user to examine a portion of a 
monitored program associated with said matching program data 
stream. 

59. a computer-readable storage medium as claimed in claim 
58, wherein said binary information directs the computer to 
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generate said data stream from data signals embedded in said 
broadcast signals . 

60. A computer-readable storage medium as claimed in claim 

59, wherein said broadcast signals are television signals 
and said binary information directs the computer to generate 
said data stream from data signals embedded in the vertical 
blanking interval between successive frames. 

61. A computer-readable storage medium as claimed in claim 

60, wherein said embedded data signals are closed caption 
signals. 

62. A computer-readable storage medium as claimed in claim 
58, wherein said binary information directs the computer to 
generate an alarm signal to alert the user when said output 
condition occurs. 

63. A computer-readable storage medium as claimed in claim 
58, wherein said binary information directs the computer to 
activate a video monitor to permit the user to watch program 
material carried by said broadcast signals when said output 
condition occurs. 

64 . A computer-readable storage medium as claimed in claim 
58, wherein said binary information directs the computer to 
cause program material contained in said broadcast signals 
to be stored in a computer file so as to permit the user to 
review the program material at a later time. 

65. A computer-readable storage medium as claimed in claim 
58, wherein said binary information directs the computer to 
store said program material in a computer file with its 
associated program data. 

66. A computer-readable storage medium as claimed in claim 
58, wherein said video monitor forms part of a multimedia 
computer system. 
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67. A computer-readable storage medium as claimed in claim 
58 , wherein said computer runs under an operating system 
employing multiple windows and said binary information 
directs the computer to display program material carried by 
said broadcast signals in a said window in accordance with 
user instructions. 

68. A computer-readable storage medium as claimed in claim 

67 , wherein said binary information directs the computer to 
display program material carried by said broadcast signals 
in a said window in response to said output condition and in 
accordance with user instructions. 

69. A computer-readable storage medium as claimed in claim 

68 , wherein said binary information directs the computer to 
display said program material in a said window in 
association with related text material generated from said 
program data stream. 

70. A computer-readable storage medium as claimed in claim 
67, wherein said binary information directs the computer to 
permit the user to view any portion of said program material 
by pointing to the associated text material in a said 
window. 

71. A computer-readable storage medium as claimed in claim 
58 , wherein said computer is a network server. 
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METHOD AND APPARATUS FOR 
AUTOMATICALLY IDENTIFYING AND 
SELECTIVELY ALTERING SEGMENTS OF A 
TELEVISION BROADCAST SIGNAL IN 
REAL-TIME 

This application is a continuation of application Ser. No. 
08/742,983 filed Nov. 1, 1996, now VS. Pat. No. 6,002,443. 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The invention relates to television signal processing sys- 
tems and in particular to television signal processing sys- 
tems for detecting selected segments of a television signal 
such as commercial advertisement segments. 

2. Description of Related Art 

It has long been a desire of many television viewers to 
have a system which automatically detects selected seg- 
ments of a television signal such as commercial advertise- 
ments or other unwanted broadcast segments in real-time 
during a television broadcast for the purpose of, for 
example, muting the audio portion of the television broad- 
cast during the unwanted broadcast material. Although a 
wide variety of techniques have been developed for detect- 
ing selected segments of television signals, heretofore there 
have been no sufficiently effective and economical systems 
that automatically operate in real-time for muting or similar 
purposes. 

Examples of broadcast segment detection systems include 
systems for detecting commercial advertisements during the 
playback of a television program on a video cassette 
recorder (VCR) for the purpose of controlling the VCR to 
skip or scan over the commercial advertisements during 
playback such that a viewer may watch the recorded tele- 
vision program without significant commercial advertise- 
ment interruptions. One particularly effective system is 
described in U.S. Pat. No. 5,333,091 entitled "Method and 
Apparatus for Controlling a Videotape Player to Automati- 
cally Scan Past Recorded Commercial Messages" and sold 
under the trademark "Commercial Advance." With the sys- 
tem of U.S. Pat. No. 5333,091, an automatic editing device 
is coupled between a conventional VCR and a conventional 
television set. As a broadcast program is recorded on a 
videotape by the VCR, the editing device detects and records 
the timing and duration of selected events, such as black 
frames combined with low audio, that separate segments of 
the broadcast and creates a time-based map of the selected 
events. After the recording is completed, the editing device 
analyzes the time-based map and determines therefrom 
which segments are commercial advertisement segments 
and which are program segments. The editing device then 
records control signals on the videotape prior to and fol- 
lowing the commercial advertisement segments. Upon 
playback, the control signals control the VCR to fast-scan 
over the commercial advertisement segments so that the 
recorded broadcast can be viewed substantially without 
interruption. Related techniques are described in U.S. Pat. 
No. 5,455,630 and in PCX application PCT/US94/00223 
published as WO 94/16442. 

Although the systems of U.S. Pat. Nos. 5,333,091 and 
5,455,630 and PCT application PCT/US94/00223 are very 
effective in skipping over commercial advertisements and 
other unwanted broadcast material during the playback of a 
recorded broadcast program on a videotape, the systems do 
not operate in real-time during a received broadcast. 

Other examples of broadcast segment detection systems 
include systems for detecting commercial advertisements 
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during the recording of a television program on a VCR for 
the purposes of controlling the VCR to skip the commercial 
advertisements during recording such that a viewer may 
later watch the recorded television program without any 
5 commercial advertisement interruption whatsoever. An 
example is a system described in U.S. Pat. No. 4,602,297 
entitled "System for Editing Commercial Messages from 
Recorded Television Braodcasts" which stores and delays 
the output of a received broadcast signal to a VCR untfl it 

10 can determine whether the delayed portion of the broadcast 
signal includes commercial advertisements. To this end, the 
system detects commercial advertisements or groups of 
commercial advertisement within the delayed portion by 
detecting amplitude drops in the broadcast signal and deter- 

15 mining whether time intervals between the amplitude drops 
correspond to normal durations of commercial advertise- 
ments or groups of commercial advertisements. The system 
then outputs only those portions of the delayed signal to the 
VCR that do not correspond to commercial advertisements 

2Q of groups of commercial advertisements. Hence, the broad- 
cast signal is recorded without commercial advertisements 
for future playback. 

Other examples of systems for detecting commercial 
advertisements during the recording of a television program 

25 are described in U.S. PaL No. 4,750,052 entitled "Apparatus 
and Method for Automatically Deleting Selected Program 
Intervals from Recorded Television Broadcasts" and U.S. 
Pat. No. 4,782,401 entitled "Editing Method and Apparatus 
for Commercials During Video Recording", both of which 

30 also operate to detect commercial advertisements based 
upon intervals occurring between amplitude drops or other 
fads in a received broadcast signal. Rather than storing and 
deleting the received signals prior to output to a VCR, 
however, the systems of U.S. Pat. Nos. 4,750,052 and 

35 4,782,401 record the signals on a video tape using a VCR 
and control the VCR to rewind at the end of each commer- 
cial advertisement or group of advertisements such that 
further program signals are recorded over the commercial 
advertisements. 

40 None of the aforementioned systems for detecting broad- 
cast segments during the recording of a television program 
operate in real-time for the purpose of, for example, muting 
the audio during commercial advertisements. 
Still other examples of broadcast segment detection sys- 

45 terns include systems for detecting commercial advertise- 
ments in broadcast television signals for the purposes of 
verifying that selected commercial advertisements are being 
broadcast on the proper days and at the proper times. 
Examples include U.S. Pat. No. 5,504,518 entitled "Method 

so and System for Recognition of Broadcast Segments" 
wherein stored digitized segments of broadcast signals are 
processed by an expert system to determine whether any 
particular segment corresponds to a selected segment of 
interest, such as a selected commercial advertisement Seg- 

55 ments that cannot be identified by the expert system, such as 
new or otherwise unknown segments, are routed to a human 
operator for verifying that the segment represents a com- 
mercial advertisement rather than a portion of a television 
program. Selected segment information such as the date and 

60 time of broadcast of the segments of interest are recorded 
and tabulated in a report for use by, for example, broadcast 
advertisers for verifying that the selected segments were 
broadcast on the proper dates and at the proper times. As part 
of its analysis of the segments, the expert system compares 

65 broadcast signatures with a database of signatures. A variety 
of techniques are described for defining and detecting sig- 
natures. The expert system also considers various cues 
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indicating the lengths and relative groupings of segments 
and the intervals therebetween. No real-time analysis 
appears to be performed by the expert system or by the 
human operator. 

Another system for commercial advertisement broadcast 
verification is described in Japanese Laid-Open Patent 
Application No. 1-174192, of Sony Corporation, entitled 
"Television Signal Processing Unit" wherein an entire days 
worth of broadcast programs are recorded on videotape. The 
recorded broadcasts are analyzed to identify commercial 
advertisement segments by intervals therebetween as 
detected by luminance signal considerations. The commer- 
cial advertisements, and only the commercial 
advertisements, are then output and recorded on a second 
videotape. The second videotape thereby contains a days 
worth of commercial advertisements in extracted form that 
may be viewed by a human operator for broadcast verifica- 
tion purposes. Again, no real-time detection of commercial 
advertisements is performed. 

Examples of systems for detecting broadcast segments, 
such as commercial advertisement segments, which have 
some real-time detection capability include U.S. Pat. Nos. 
4,420,769 entitled "Device for the Automatic Editing, at the 
Receiver, of Unwanted Program Material from Broadcast 
Electrical Signals" and 4,750,213 entitled "Method and 
System for Editing Unwanted Program Material from 
Broadcast Signals". U.S. Pat. No. 4,420,769 describes a 
system wherein a human operator identifies program seg- 
ments to be eliminated from a broadcast signal causing a 
digital code word or signature for the segment to be stored 
in memory. Subsequently, if the same segment is 
encountered, the system automatically compares digitized 
portions of the received signal with the stored signatures to 
recognize the segment and then to blank out or otherwise 
eliminate the segment in real-time. One significant problem 
with this system is that a human operator must first detect 
program segments to be eliminated. Also, a significant 
amount of data processing is required to establish and 
compare the signatures in a reliable manner. U.S. Pat. No. 
4,750,213, by the same inventor, sets forth improvements 
intended, in part, to reduce the processing time required. The 
system of U.S. Pat. No. 4,750,213 also includes a delay or 
other storage device for allowing a received signal to be 
delayed until a determination can be made by the human 
operator as to whether a particular segment of the received 
signal should be eliminated. If so, the received, delayed 
signal is output with the selected blanked out or otherwise 
eliminated. Although U.S. Pat No. 4,750,213 may represent 
an improvement over U.S. Pat. No. 4,420,769, it shares the 
disadvantage that a human operator must detect program 
segments to be eliminated. 

Another example of a system which has some real-time 
detection capability is U.S. Pat. No. 5,151,788 entitled 
"Method and Apparatus for Identifying and Eliminating 
Specific Material from Video Signals'* wherein commercial 
advertisement broadcast segments are distinguished from 
program segments by the relative level of "activity" within 
the beginning of each segment. If the activity level is high, 
the segment is presumed to be a commercial advertisement. 
The system detects the amount of activity within the first few 
seconds of a segment following a blank frame and operates 
to mute or otherwise eliminate the remaining portions of the 
segment if the detected activity level is representative of a 
commercial advertisement. The system may control a VCR 
to stop recording for the remainder of the segment. The 
patent describes that the optimum time period during which 
the activity level should be detected is 8 seconds. One 
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disadvantage of the system is that the detection of commer- 
cial advertisements based upon an activity level is likely to 
be unreliable for many broadcasts and, even when the 
system does correctly detect a commercial advertisement, 

5 that detection does not occur until 8 seconds into the 
commercial advertisement 

Other references that relate to systems for detecting 
selected segments of a broadcast signal such as commercial 
advertisement segments of a television signal are described 

to in the following references: U.S. Pat. No. 3,919,479 entitled 
"Broadcast Signal Identification System"; 4,229,765 entitled 
"Remote Audio and Brightness Control System for a Tele- 
vision"; 4,230,990 entitled "Broadcast Program Identifica- 
tion Method and System"; 4,259,689 entitled "Television 

15 Advertising Editing System"; 4333,110 entitled "Television 
Editing System"; 4,390,904 entitled "Automatic Circuit and 
Method for Editing Commercial Messages from Television 
Signals"; 4,667,466 entitled "Broadcast Program Identifica- 
tion Method and Apparatus"; 4,697,209 entitled "Methods 

20 and Apparatus for Automatically Identifying Programs 
Viewed or Recorded"; 4,739398 entitled "Method, Appa- 
ratus and System for Recognizing Broadcast Segments"; 
4,752,834 entitled "Reciprocating Recording Method and 
Apparatus for Controlling a Video Recorder so as to Edit 

25 Commercial Messages from a Recorded Television Signal"; 
5,019,899 entitled "Electronic Data Encoding and Recog- 
nition System" and European Patent Application Publication 
No. 0161512 entitled "Program Identification System". 
Hence various techniques have been developed for detect- 

30 ing selected broadcast segments such as commercial adver- 
tisements within a television signal. None of the techniques 
described above, however, are effective for reliably and 
automatically detecting broadcast segments in real-time 
without the need of a human operator. According, it would 

35 be desirable to provide a simple, inexpensive system which 
reliably detects selected segments of a broadcast signal 
without requiring a human operator and wherein, either 
immediately or after some period of time, all or most 
selected segments can be detected in real-time. 

40 SUMMARY OF THE INVENTION 

In accordance with the invention, a method and apparatus, 
intended to be marketed under the mark SoundlOff™, is 
provided for identifying selected segments of a broadcast 

45 signal in real-time for the purpose of, for example, muting 
the video and audio portions of the signal during the selected 
segment. Hereinafter, "selected segment" refers to any 
selected portion of a broadcast signal including, for 
example, unwanted broadcast segments such a commercial 

50 advertisement segments. In one embodiment wherein the 
broadcast signal is a television signal, the method includes 
the steps of detecting a signature pattern associated with the 
beginning of a received segment of the television signal and 
comparing the signature pattern to stored signature patterns 

55 representative of selected segments. If the signature pattern 
matches one of the stored signature patterns, the received 
segment is thereby immediately identified as being one of 
the selected segments and can be processed accordingly in 
real-time to, for example, mute the audio and video portions 

60 of the television signal during the received segment as the 
television signal is output to, for example, a television 
monitor. If the signature pattern of the received segment 
does not match any of the stored signature patterns, the 
received segment is analyzed to determine whether it is 

65 nevertheless one of the selected segments and, if so, its 
signature pattern is stored along with the stored signature 
patterns. The analysis to determine whether the received 
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segment is one of the selected segments even though its 
signature does not match any of the stored signatures may be 
performed by, for example, detecting the length of the 
received segment upon its completion and then determining 
whether the length of the received segment matches any of 5 
a pre -determined set of permissible lengths for selected 
segments. If so, the segment is identified as being a selected 
segment and its signature pattern is stored along with the 
other stored signature patterns such that, the next time the 
same segment is received, its signature will then match one 10 
of the stored signatures and therefore the received segment 
will be identified promptly at a selected segment and muted 
or otherwise altered in real-time. For an embodiment 
wherein the selected segments are commercial advertise- 
ment segments, the permissible lengths are, for example, 15 ^ 
seconds, 30 seconds, and 60 seconds. The method may be 
performed by a television set or other device receiving a 
television signal such as a personal computer television 
(PCTV). 

In one implementation, no stored signatures are initially 20 
provided. Hence, the first time any particular selected seg- 
ment is encountered, it is not immediately identified as being 
a selected segment and therefore is not muted in real-time. 
However, the next time the same segment is encountered, 
the segment will be subject to immediate real-time muting 25 
because its signature will have been stored the first time it 
was received. Thereafter, as long as its signature remains 
stored, the segment will be subject to real-time muting 
whenever it is received. As more and more selected seg- 
ments are encountered and their respective signature pat- 30 
terns stored, the more and more likely it becomes that any 
particular selected segment encountered while watching a 
television program will be promptly detected based upon its 
signature and muted in real-time. Hence, even though no 
stored signatures are initially provided, a database of sign a- 35 
tures is soon developed. Also, in the exemplary 
implementation, the selected segment detection method may 
be performed by a television set even while the television set 
is otherwise not in use to thereby detect and record segment 
signatures in a background mode to quickly develop a 40 
database of signatures. Alternatively, a database of the 
signatures of common segments, such as common commer- 
cial advertisement segments, may be initially provided at the 
time of manufacture such that the common segments are 
thereby subject to immediate real-time muting even the first 45 
time received. 

For many television broadcast systems, commercial 
advertisements appear in groups of two or more. When used 
in connection with such systems, the method of the inven- 
tion preferably operates to not only mute or otherwise alter 50 
any detected commercial advertisements of a commercial 
advertisement group but also to mute or otherwise alter other 
unwanted broadcast segments appearing within the group, 
such as station breaks, news bulletins, public service 
announcements, etc. To this end, the method operates to 55 
identify the first of a group of commercial advertisements, 
then mutes or otherwise alters all segments during a prede- 
termined minimum commercial advertisement group time 
period. The minimum group time period may be, for 
example, two minutes and twenty seconds. Hence, all seg- 60 
meats appearing during the period are muted in real-time, 
including commercial advertisements for which the corre- 
sponding signature has not yet been stored (and which 
therefore would not otherwise be subject to immediate 
real-time muting) and segments that do not match the 65 
predetermined permissible lengths (and which therefore 
would not otherwise be subject to any muting whatsoever), 
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such as the aforementioned station breaks, news bulletins, 
and public service announcements. Thus the method is not 
limited to muting only those segments that have matching 
stored signatures. Rather, the method operates to mute or 
otherwise alter all segments appearing in a detected com- 
mercial advertisement group. However, only the signatures 
of segments having lengths matching the permissible 
selected segment lengths are stored. The signatures of a 
non-commercial advertisement segment, such as a station 
break, news bulletin, or public service announcement, are 
stored only if the length of the segment happens to match 
one of the predetermined permissible segment lengths. 

Depending upon the implementation, the method identi- 
fies the beginning of a received segment by detecting 
pre-determined event markers which separate segments. 
When implemented for use in North America, the method 
may, for example, identify a period of one or more black 
frames combined with low audio as being an event marker 
representative of the beginning of a segment. When imple- 
mented for use in Europe, the method may further identify 
periods of one or more colored frames, such as blue frames, 
combined with low audio as being an event marker. When 
implemented for use in Japan, the method may instead 
identify a frame break combined with a period of low audio 
as being an event marker. In any case, once an event marker 
denoting the beginning of a broadcast segment is detected, 
the method promptly detects a signature from the segment 
for comparison against stored signatures. The signature may, 
for example, be a binary hash code representative of whether 
selected lines of a selected frame near the beginning of the 
segment exceed pre-determined luminance threshold levels. 
By detecting the signature near the beginning of the 
segments, the signature can be compared against stored 
signatures to allow the segment to be muted or otherwise 
altered in real-time before much of the received segment has 
been displayed. Preferably, the signature is based on frames 
appearing during the first one second of the segment. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The features, objects, and advantages of the present 
invention will become more apparent from the detailed 
description set forth below when taken in conjunction with 
the drawings in which like reference numerals identify 
correspondingly throughout and wherein: 

FIG. 1 is a block diagram illustrating a television set 
incorporating a commercial advertisement detecting and 
muting system configured in accordance with an exemplary 
embodiment of the invention for detecting a commercial 
advertisement or other selected broadcast segment in real- 
time during a television broadcast and for muting video and 
audio portions of the television signal during the commercial 
advertisement; 

FIG. 2 is a timing diagram illustrating exemplary group- 
ings of consecutive commercial advertisement segments 
within a television broadcast signal analyzed by the system 
of FIG. 1 and illustrating event markers separating the 
broadcast segments; 

FIG. 3 is a timing diagram illustrating the muting of 
luminance and audio portions of a television signal provided 
by the system of FIG. 1 for a pair of broadcast segments both 
identified as being commercial advertisements; 

FIG. 4 is a timing diagram illustrating exemplary prede- 
termined points in time following a first event marker 
wherein the system of FIG. 1 searches for a second event 
marker for the purpose of determining whether the first 
event marker corresponds to the beginning of a commercial 
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advertisement segment and particularly illustrating a FIG. 16 is a diagram similar to that of FIG. 14 but wherein 

example wherein a second event marker is found at one of the database is configured to store the signature and broad - 

the predetermined points in time thereby indicating that the cast date/time information for each detected broadcast of 

first event marker corresponds to the beginning of a com- each commercial advertisement to thereby allow verification 

mercial advertisement segment; 5 that selected commercial advertisement are being broadcast 

FIG. 5 is a timing diagram similar to that of FIG. 4, but on the proper days and at the proper times; 

wherein a second event marker is not found until after all of FIG. 17 is a flowchart illustrating a method for operating 

the predetermined points in time thereby indicating that the the system of FIG. 1 wherein the system is operated even 

first event marker does not correspond to the beginning of a while the television set is otherwise not in use to thereby 

commercial advertisement segment; 10 detect and record commercial advertisement segment sig- 

F1G. 6 is a timing diagram similar to that of FIG. 4, but natures in a background mode to thereby quickly develop a 

wherein a second event marker is found but not at one of the full database of signatures; 

predetermined points in time thereby also indicating that the FIG. 18 illustrates a system wherein an apparatus per- 

first event marker does not correspond to the beginning of a forming the method described with reference to FIGS. 1-17 

commercial advertisement segment; 15^ provided external to a television set; and 

FIG. 7 is a timing diagram similar to that of FIG. 4, Ha 19 ilhlstrates a pcjv configured with hardware or 
wherein a second event marker is found at one of the for donalD me method described ^ refer . 
predetermined points in time thereby also indicating that the ence tQ pics 
first event marker corresponds to the beginning of a com- 
mercial advertisement segment but wherein yet another 20 DETAILED DESCRIPTION OF EXEMPLARY 
event marker, which must be ultimately ignored, is found EMBODIMENTS 
between the otters; Referring to the figures, exemplary embodiments of the 

!? Jl 3 1° *YT P i Jy m Mention will be described with reference to a system 

performed by the system of FIG. 1 while the television set rimaril directed towaid detecting commercial advertise- 

is in operation for detecting and I muting commercial adver- * ^ fa ided ^ reference to 

tisements during a television broadcast wherein, among me block dia ^ Umin ^ of nGS U17 

other features, a signature corresponding to an event marker ^ of ^ ££ lcmcnmioos m pro . 

is compared against a database of signatures corresponding ^ ^ reference to tbe flowcbarts and block of 

to known commercial advertisements to determine whether nGS g_ 19 Xhe exemplary embodiments are priinarily for 

the event marker corresponds to the beginning of a com- 30 ^ {n afld mutiflg commerciaJ advertisements in a 

mercial advertisement segment; North television broadcast signal wherein the 

FIG. 9 is a flowchart illustrating steps performed in advertisements are of fixed predetermined lengths and 

furtherance of the method of FIG. 8 to read a segment appear m As will be described, the system can 

signature corresponding to an event marker, ^ addilionaUy mute olher unwanted program material seg- 

FIG. 10 is a representation of a single exemplary test menls inching station breaks, news bulletins etc. that 

frame of a television broadcast signal analyzed during appear in a detected commercial advertisement group, 

execution of the steps of FIG. 9 and in particular illustrating Moreover, the invention may be alternatively implemented 

exemplary signature test lines of the frame from which the f or ^ with other television broadcast systems, such as 

signature is determined; ^ those of Europe and Japan, and may be alternatively imple- 

HG. 11 is a representation of an exemplary bit string mented to identify other types of selected segments of a 

binary hash code signature corresponding to the frame of signal besides commercial advertisements. 

FIG. 10 and in particular showing exemplary bit values each p, G x illustrates a television set 100 configured with 

representative of whether the average luminance of a cor- Sound! Off™ and having a receiver 102 and a monitor 104. 

responding test line of the frame exceeds a preKletermined 45 Television receiver 102, which includes a television tuner, 

threshold. receives an input broadcast television signal from an antenna 

FIG. 12 is a flowchart illustrating steps performed in 106 or from any other appropriate input source such as a 

furtherance of the method of FIG. 8 to determine if an event coaxial cable input (not separately shown) and outputs a 

marker nevertheless corresponds to the beginning of a received television signal for eventual display by monitor 

commercial advertisement segment even through the signa- 50 104 i n this exemplary implementation, the received televi- 

ture corresponding to the event marker does not match any s j on signal is an analog NTSC signal including audio, 

of the signature of the database; luminance and chrominance components. In other 

FIG. 13 is a flowchart illustrating steps performed in implementations, the television signal is a PAL or SECAM 

furtherance of the method of FIG. 8 to store a signature in signal or digital signal such as an HDTV signal. Monitor 104 

the database after it has been determined that the signature 55 displays the video portions of the television signal and plays 

corresponds to a commercial advertisement segment; the audio portions through speakers (not separately shown). 

FIG. 14 is a representation of a signature database Rather than routing the received television signal directly 

accessed during execution of the steps of FIG. 13 wherein to the monitor for display, the television signal is first routed 

the database is configured as a RAM storing a single record through a commercial advertisement detecting and muting 

of information, including broadcast date/time information, 60 system 108 which operates in real-time to detect target or 

for each unique commercial advertisement signature to selected segments of tbe television signal and to mute 

thereby allow records to be selectively overwritten as selected components of the television signal during those 

needed based, for example upon the broadcast date/time segments prior to output to monitor 104. In the exemplary 

information. implementation, the selected segments are commercial 

FIG. 15 is a diagram similar to that of FIG. 14 but wherein 65 advertisements. The muted components of the television 

the database is configured as a first-in first-out (FIFO) signal are the audio and luminance components. As noted, in 

queue; other implementations, other selected segments may be 
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detected or other components of the signal may be muted or 
otherwise processed. Commercial advertisement detecting 
and muting system 108 is, in the exemplary implementation, 
configured as a single plug-in module having both analog 
and digital electronics for plugging directly into an other- 5 
wise conventional television set. In other implementations, 
described below, the commercial advertisement detecting 
and muting system is an after-market device configured for 
use with a television set having audio and video outputs or 
is hardware or software provided within a PCTV. 10 

Detection of commercial advertisement segments is 
achieved as follows. Within detecting and muting system 
108, the television signal is first routed through a broadcast 
segment event marker and signature detector 110 which 
detects event markers separating broadcast segments of the 15 
television signal and, for each segment, detects a signature. 
A broadcast segment may be a commercial advertisement 
segment or a portion of a television program, movie, news 
show, "infomercial" etc. hereinafter referred to as program 
segments. As noted, the broadcast segments are separated by 20 
event markers. In the exemplary implementation which is 
for use in North America, an event marker is a period of one 
or more black frames combined with low audio. When 
implemented for use in Europe, the event marker may also 
be a period of one or more colored frames, such as blue 25 
frames, combined with low audio. When implemented for 
use in Japan, an event marker is a frame break combined 
with a period of low audio extending from slightly before to 
slightly after the frame break. 

In any case, commercial advertisement segments may be 30 
distinguished from program segments in that commercial 
advertisement segments almost always appear in groups of 
two or more and are each of one or more relatively short 
pre-determined lengths such as 15 seconds, 30 seconds and 
60 seconds. The groups are typically at least two minutes 35 
long. FIG. 2 illustrates an exemplary television signal hav- 
ing program segments 112, 114, and 116 followed by groups 
of commercial advertisement segments 118, 120 and 122, 
respectively, with each segment separated by an event 
marker 124. Commercial advertisement group 118 includes 40 
two commercial advertisement segments of 60 seconds 
each. Group 120 includes two commercial advertisement 
segments of 30 second and one of 60 seconds. Group 122 
also includes two 60 second commercial advertisement 
segments. Although not shown in FIG. 2, commercial adver- 45 
tisement groups may additionally include other segments 
such as station breaks, news bulletins, public service 
announcements etc., which are typically not of the same 
lengths as commercial advertisements, i.e. the other seg- 
ments are not 15, 30 or 60 seconds long. If other segments 50 
are included, the group is typically at least two minutes and 
twenty seconds long. The manner by which the non- 
commercial advertisement segments within a group are 
processed will be described below. 

Each unique broadcast segment has a unique signature. 55 
The signature may be any unique characteristic of the 
segment that can be used to distinguish one segment from 
another. As will be described blow, in the exemplary 
implementation, the signature is a bit string hash code 
representative of whether each of a set of selected lines of a 60 
selected frame of a received segment has an average lumi- 
nance level that exceeds a predetermined threshold. The 
selected frame may be, for example, the tenth frame fol- 
lowing the completion of the event marker marking the 
beginning of the segment. The selected lines may be, for 65 
example, the first 64 odd lines of the frame beginning at line 
23. Other signatures may alternatively be employed. 
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Referring again to FIG. 1, as each broadcast segment is 
received, the signature for that broadcast segment is detected 
in real-time by detector 110. The signature is routed to a 
control unit 126 which accesses a signature memory unit 
128 to determine whether the detected signature matches 
any of a set of signatures stored therein known to correspond 
to commercial advertisement segments. If the detected sig- 
nature matches one of the stored signatures, then the control 
unit identifies the broadcast segment from which the signa- 
ture was taken as being a commercial advertisement seg- 
ment and forwards a mute control signal to a video/audio 
mute unit 130. The mute unit also receives the television 
signal directly from detector 110. In response to the mute 
control signal, the mute unit mutes the audio and luminance 
components of the television signal during the commercial 
advertisement segment and forwards the muted signal to 
monitor 104. The mute control signal is maintained during 
the entire commercial advertisement segment. In the exem- 
plary implementation, the signature memory unit stores, for 
each commercial advertisement signature, the length or 
duration of the segment The control unit reads this infor- 
mation and thereby knows when to terminate muting. In 
other implementations, muting is terminated upon detection 
of a next event marker. The former technique is preferred 
because event markers may coincidentally appear in the 
middle of a commercial advertisement segment. If muting is 
always terminated upon detection of a next event marker, 
muting could erroneously be removed in the middle of a 
commercial advertisement. 

Hence, the system mutes the output signal during a 
detected commercial advertisement segment. Additionally, 
the system may continue the mute throughout an entire 
commercial advertisement group containing the detected 
commercial advertisement segment. To this end, the system 
determines when the commercial advertisement group 
began and continues muting for at least some minimum 
predetermined time period, such as two minutes, following 
the beginning of the group. In this manner, non-commercial 
advertisements of the group are muted as well as any 
commercial advertisements of the group not having a pre- 
viously stored signature. The manner by which the system 
determines the beginning of a commercial advertisement 
group for the purposes of muting the entire group will be 
described below. 

By "muting" it is meant that the amplitudes of the audio 
and luminance components of the output signal are reduced 
such that the viewer of monitor 104 sees a dimmed image 
and hears reduced audio levels. FIG. 3 is a timing diagram 
illustrating the muting of luminance 132 and audio 134 
components of a television signal during two consecutive 
detected commercial advertisement segments between a pair 
of un -muted program segments. As can be seen, the levels of 
both the audio and luminance components are ramped down 
immediately after the beginning of the first commercial 
advertisement segment and are ramped up immediately after 
the end of the second commercial advertisement segment. 
Mute unit 130 (FIG. 1) also superimposes an audio chime 
over the audio signal as the audio signal is increased to its 
p re-mute level. The chime is provided to alert a viewer mat 
the muted commercial advertisement segments have com- 
pleted. This is particularly desirable if the next program 
segment begins with a silent audio track, perhaps accompa- 
nied by program credits. Without the chime, the viewer, 
whose attention may be elsewhere, might not realize that a 
program segment of interest is beginning. In other 
implementations, the chime may also be produced at the 
beginning of each muted commercial advertisement seg- 
ment. 
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Referring again to FIG. 1, if the signature detected by Accordingly, the system searches for subsequent event 

detector 110 does not match any of the stored signatures of markers within a range of time centered around each 

memory unit 128 and the system is not currently muting an expected event marker time. For a 30 second commercial 

entire commercial advertisement group, then no mute con- advertisement, the system may search, for example, for 

trot signal is provided to mute unit 130 and the television 5 event markers occurring 28.9 to 32 seconds following a 

signal is thereby forwarded without modification by the previous event marker. 

mute unit to monitor 104. In this manner, program segments In any case, if a determination is made that a broadcast 

remain un-muled. More specifically, when a program seg- segment is a commercial advertisement, then the next time 

ment is received, its signature is detected and compared the same commercial advertisement segment is received the 

against those of memory unit 128. Because the memory unit 1Q following occurs. The signature of the commercial adver- 

only stores segments corresponding to commercial adver- tisement segment is again detected by detector 110 and 

tisement segments or other selected broadcast material, the compared against those of memory unit 128. The newly 

program segment signature will not match any stored seg- detected signature matches the previously stored signature 

ment and no muting will therefore occur during the program and the commercial advertisement segment is therefore 

segment. 15 muted in real-time. In this manner, the system builds a 

Hence, commercial advertisement detecting and muting database of commercial advertisement segment signatures 

system 108 operates to mute in real-time any broadcast and each commercial advertisement segment is typically 

segment having a signature matching one of the signatures displayed only once in its un-muted form. Thereafter, as 

of memory unit 128. It is, of course, possible that a com- long as the signature of the commercial advertisement 

mercial advertisement will be received for which the sagna- ^ segment remains in the database, the segment is detected and 

ture had not previously been recorded in memory unit 128 muted in real-time. Hence, even if the signature database of 

and the commercial advertisement will therefore not be memory unit 128 is initially empty, a database of com me r- 

muted (unless it is part of an entire group of commercial cial advertisement signatures is developed and soon all or 

advertisements being muted). Indeed, in the exemplary most commercial advertisements that are commonly shown 

implementation, when the system is initially activated, the ^ are detected and muted. Of course, it is possible for a 

memory unit has no signatures whatsoever stored therein. non-commercial advertisement segment of a desired televi- 

Accordingly, detecting the muting system 108 is provided sion program to be 15, 30 or 60 seconds long such that its 

with a non-real-time means for determining whether a signature will be added to the database along with those of 

broadcast segment, having a signature that does not match commercial advertisements. Hence, the next time the same 

any signature recorded in the database, nevertheless is a ^ television program is received the segment will be muted 

commercial advertisement segment. Once a determination is under the assumption that it is a commercial advertisement, 

made that a broadcast segment is a commercial advertise- This is not a particularly significant problem. The viewer 

ment segment or other selected segment, the signature of the merely over-rides the mute and continues to view the 

segment is added to the database such that the segment can program. Moreover, television programs are not typically 

be muted in real-time the next time it is encountered. 35 viewed very frequently and, as will be described below, the 

The determination of whether a broadcast segment is a system may be configured to erase signatures not frequently 

commercial advertisement segment or other selected encountered such that, the next time the same television 

segment, assuming that its signature does not match a stored program is viewed, the signature of that segment of the 

signature, is achieved as follows. As noted above, and as program may have already been erased. Also, it is possible 

illustrated in FIG. 2, commercial advertisement segments 40 for news bulletins, public service announcements etc. to also 

are distinguishable from program segments in that commer- be 15, 30 or 60 seconds long such that their signatures will 

cial advertisement segments almost always appear in groups be added to the database for future muting. Again, this is not 

of two or more and are each of one or more relatively short a significant problem. Indeed, if the news bulletins, public 

pre -determined durations such as 15 seconds, 30 seconds service announcements etc. are broadcast frequently, the 

and 60 seconds. Accordingly, if an event marker corresponds 45 viewer may consider their subsequent muting to be quite 

to the beginning of a commercial advertisement segment, a advantageous. 

second event marker should occur after one of the pre- As noted, a determination of whether a segment consti- 

detenmned periods of time following the first event marker. tutes a commercial advertisement segment even though its 

If an event marker corresponds to the beginning of a signature does not match a stored signature is performed 

program segment, a second event marker will not likely 50 based upon the time periods separating event markers. FIGS, 

occur after one of the pre-determined periods of time fol- 4-7 illustrate various example of event marker groupings 

lowing the first event marker. within a television signal to help illustrate the manner by 

Hence, if a second marker occurs, for example, 15, 30 and which this determination of a commercial advertisement 

60 seconds following a first event marker, a determination is segment is performed. 

made that the segment between the first and second event 55 FIG. 4 illustrates an example wherein a second event 

markers is therefore probably a commercial advertisement marker occurs at one of the pre-determined time periods 

or other selected segment and the signature detected follow- following a first event marker. More specifically, FIG. 4 

ing the first event marker is added to the signatures stored in shows a pair of event markers 133 and 135 occurring 60 

memory unit 128 (FIG. 1). If a second marker does not occur second apart. Shortly after first event marker 133 is detected, 

at 15, 30 or 60 seconds following a first event marker, a 60 the signature for the program segment beginning at that 

determination is made that the segment between the first and event marker is temporarily recorded. Then, a determination 

second event markers is therefore probably not a commer- is made as to whether a second event marker occurs at 15, 

cial advertisement segment and the signature detected fol- 30 or 60 seconds following the first event marker. In the 

lowing the first event marker is merely discarded. example of FIG. 4, second marker 135 is detected 60 

In practice, for North American broadcast systems, the 65 seconds from the first marker. Accordingly, shortly 

lengths of commercial advertisements often deviate slightly thereafter, the temporarily recorded signature is saved i.e. it 

from the standard 15, 30 and 60 second durations. is stored in memory unit 128 (FIG. 1) along with any other 
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pertinent information such as the duration of the correspond- another event marker 148 occurring between event markers 

ing commercial advertisement as determined by the time 144 and 146 but not at one of the predetermined time 

period between the first and second event markers. If the periods. The detection of the additional event marker in the 

second marker had alternatively occurred at either 15 or 30 interim between event markers 144 and 146 does not ter- 

seconds following the first event marker, the signature 5 minate the search for an additional event marker at any of 

would have been saved then instead. the predetermined time periods from the first event marker. 

With further reference to FIG. 4, it should be noted that The detection of event marker 148 however triggers a 
the detection of second event marker 135 also triggers a temporary recording of another signature immediately there- 
temporary recording of the signature of the segment begin- after and a contemporaneous search for event markers 
ning at the second event marker. This signature is compared 10 following at 15, 30 or 60 second intervals, 
with those stored in the memory unit (FIG. 1). As with the Hence, FIGS. 6-7 illustrate that more than one signature 
first signature, if a match is detected then the broadcast mav be temporarily recorded at any given time and that 
segment beginning at the second event marker is immedi- amplitude contemporaneous searches may be performed for 
ately muted and if no match is detected, then the signature subsequent event markers with one search for each event 
is temporarily recorded until a determination can be made 15 mar ker. Indeed, in the exemplary implementation, each 
whether the broadcast segment beginning at the second event marker, for which the corresponding signature is not 
event marker is nevertheless a commercial advertisement found in the signature database, triggers a search for an 
segment. additional event marker at one of the pre-determined inter- 

FIG. 5 illustrates an example wherein a second event vals thereafter. As each search may last as long as 60 seconds 

marker does not occur at one of the pre-determined time 20 during which time a number of additional event markers 

periods following a first event marker. More specifically, may be encountered, a number of signatures may need to be 

FIG. 5 shows a pair of event markers 136 and 138 occurring simultaneously temporarily stored and coordinated, 

more than 60 seconds apart Shortly after first event marker Additionally, the time that each event marker is detected is 

136 is detected, the signature for the broadcast segment recorded to allow calculation of the various time intervals, 

beginning at that event marker is temporarily recorded. 25 Accordingly, a temporary signature and event marker time 

Then, a determination is made as to whether a second event memory unit 150 is provided in connection with control unit 

marker occurs at 15, 30 or 60 seconds following the first 126 of FIG. 1 for storing the temporary signatures and the 

event marker. In the example of FIG. 5, no event marker is time of the event marker corresponding to each temporarily 

detected at either 15, 30 or 60 seconds following the first stored signature. Signatures stored therein are either erased 

event marker. Accordingly, shortly after the time of event 30 after 60 seconds, if no event marker is detected at any of the 

marker 136 plus 60 seconds, the temporarily recorded sig- pre-determined intervals, or transferred to the signature 

nature is discarded (Again it should be noted that the memory unit 128, if an event marker is detected at any of the 

detection of the second event marker also triggers a tempo- pre-determined intervals. In the exemplary embodiment, 

rary recording of the signature of the segment beginning at control unit 126 of FIG. 1 coordinates storage of information 

the second event marker and processing of that signature 35 within temporary memory unit 150 and performs the various 

proceeds.) searches for additional event markers. In this regard, the 

FIG. 6 illustrates an example where a pair of first and control unit may be a microprocessor pre-programmed with 
second event markers 140 and 142 are separated by less than appropriate software to coordinate the aforementioned 
60 seconds but where second event marker 142 does not operations. Alternatively, the control unit may be a dedicated 
occur at one of the pre-determined intervals. Hence, the first 40 integrated circuit pre-configured with appropriate hard- 
marker does not correspond to the beginning of a commer- wired logic to coordinate the operations. Other appropriate 
cial advertisement or other selected segment. It should be devices may be employed as well. 

noted that the determination that the first marker does not Thus, by detecting various event marker times, the system 

correspond to a commercial advertisement is not made until can determine whether a segment, which does not have a 

60 seconds following the detection of the first event marker. 45 matching stored signature, is nevertheless a commercial 

Accordingly, the temporarily recorded signature recorded advertisement segment. Hence, each received commercial 

following the first event marker is not discarded until after advertisement is detected either immediately via a signature 

the 60 second mark. Hie detection of the second event match or at the end of the segment via an event marker 

marker in the interim does not terminate the search for analysis. Once a commercial advertisement is detected, the 

additional event markers based upon the first event marker. 50 system determines whether it represents the beginning of a 

That search continues until at least 60 seconds from the first new commercial advertisement group and, if so, muting is 

event marker. Also, the detection of the second event marker maintained for at least two minutes thereafter, regardless of 

triggers a temporary recording of a second signature and a whether the subsequent segments have matching signatures 

contemporaneous search for additional event markers fol- in the database. In one implementation, this is achieved as 

lowing the second event marker at 15, 30 or 60 second 55 follows. For any detected commercial advertisement 

intervals. segment, if no other commercial advertisement segment 

FIG. 7 illustrates an example where a pair of event preceded it during the previous 60 seconds, or some other 

markers 144 and 146 are separated by 30 seconds. Hence, predetermined minimum period of time, then the system 

the first marker corresponds to a commercial advertisement, identifies the detected commercial advertisement segment as 

and a signature temporarily recorded shortly after the first 60 being the first of a new group. If it were not the first of a new 

event marker is stored along with other pertinent informa- group, then a previous commercial advertisement segment 

tion in the database of memory unit 128 (FIG. 1). It should of the same group would at least have terminated at some 

be noted that the determination that the first and second point during the preceding 60 seconds. (This assumes that no 

event markers 144 and 146 correspond to the beginning and two consecutive commercial advertisement segments of a 

end of a commercial advertisement is made shortly after 65 common group are ever separated by more than 60 seconds 

second event marker 146 is detected and is not deferred until of non-commercial advertisement segments such as station 

60 seconds from the first event marker. Also, FIG. 7 shows breaks etc. In most cases, this assumption is correct.) Once 
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the system identifies the first commercial advertisement of 
the group, muting is initiated and maintained for at least two 
minutes, or some other predetermined minimum group rime 
period, from the beginning of that commercial advertise- 
ment Notably, this determination is made regardless of 5 
whether the commercial advertisement was detected via 
signature match or event marker analysis. If detected by 
signature match, the determination is made during the recep- 
tion of the commercial advertisement If detected via event 
marker analysis, the determination is not made until the end 10 
of the segment. In either case, the duration of continuous 
muting is calculated based on the beginning of the first 
commercial advertisement segment of the group. As noted, 
muting is maintained for some predetermined period of time 
representative of the minimum duration of a group of 15 
commercial advertisements such as two-minutes. Any par- 
ticular group may be longer in duration and continuous 
group-based muting may therefore end during the group. 
Signature-based muting, however, will continue for any 
remaining commercial advertisements of the group that have ^ 
matching signatures in the database. Hence, the remaining 
advertisements of the group will not necessarily be 
un-muted. Other techniques for detecting the beginnings and 
predicting the endings of commercial advertisement groups 
may alternatively be employed. ^ 

With reference to the remaining Figures, further details of 
exemplary and alternative method and apparatus embodi- 
ments of the invention will be described. 

FIG. 8 is a flowchart summarizing steps performed by the 
commercial detection and muting system of FIG. 1 or by any 30 
other appropriate system for performing commercial adver- 
tisement detection and muting. As many of these steps have 
already been describe above, only additional aspects of the 
detection and muting technique will be described in detail. 
Initially at step 200, a television broadcast signal is received. 3s 
The television signal is searched to detect event markers, 
step 202. If an event marker is not detected, step 204, the 
television signal is merely output at step 206 perhaps to a 
display device such as monitor 104 (FIG. 1). Steps 200, 202, 
204 and 206 are repeated until an event marker is detected 40 
at step 204 triggering the beginning of a broadcast segment 
evaluation process for the broadcast segment beginning at 
the detected event marker, step 208. As will be described 
below, the broadcast segment evaluation process operates to 
determine whether the broadcast segment beginning at the 45 
corresponding event marker is a commercial advertisements 
and if so mutes the output of the television signal during the 
commercial advertisement. After the broadcast segment 
evaluation process has been triggered for that event marker, 
step 206 is again executed to continue outputting the tele- 50 
vision signal. Hence, the television signal is not stored or 
otherwise delayed but is immediately output in real-time. 
The broadcast segment signature evaluation process is per- 
formed shortly after detection of the event marker such that 
very little of the television signal is output before a deter- 55 
mination of whether the signal needs to be muted is 
achieved. 

Steps 220-206 are repeatedly performed to search the 
television signal for additional event markers. Each newly 
detected event marker triggers a separate broadcast segment 60 
signature evaluation process for the broadcast segment 
beginning at that event marker. If part of the system per- 
forming the method of FIG. 8 is implemented as a software 
program, the individual broadcast segment evaluation pro- 
cesses may be, for example, separate and distinct software 65 
processes running on a microprocessor or may merely be 
separate logic threads within a single software process. They 
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are described herein as separate processes primarily to help 
clarify that contemporaneous searches for subsequent event 
markers may need to be performed for each of a set of 
consecutive event markers and not to imply any particular 
architecture by which the method of FIG. 8 must be imple- 
mented. 

Beginning at step 210, one broadcast segment evaluation 
process triggered at step 208 is set forth. Step 210 is shown 
connected to step 208 by a dashed line to indicate that the 
broadcast segment evaluation process proceeds indepen- 
dently of, and contemporaneously with, steps 200-206. At 
step 212, the signature corresponding to the current broad- 
cast segment is read from the television signal. Additional 
details of step 212 are step forth in FIGS. 9-11 and will be 
described below. At step 214, the signature is compared 
against stored signatures in a database of signatures known 
to correspond to commercial advertisements or other 
unwanted broadcast material. If there is no signature match, 
step 216, then step 218 is performed to determine whether 
the current broadcast segment nevertheless comprises a 
commercial advertisement segment and, if so, what the 
length of the commercial advertisement segment is in sec- 
onds. Additionally details of step 218 are set forth in FIG. 12 
and will be described below. If the current segment is 
determined to not be a commercial advertisement at step 
220, then step 222 is executed to end the broadcast segment 
evaluation process. If, however, the current segment is 
determined to be a commercial advertisement, step 224 is 
executed to store the signature of the current segment read 
at step 212 in the database such that, when the same 
commercial advertisement segment is encountered again, 
the commercial advertisement can be detected in real-time. 
As noted above, the determination of whether a broadcast 
segment comprises a commercial advertisement segment 
may take up to 60 seconds from the event marker defining 
the beginning of the segment. Additional details of step 224 
are set forth in FIGS. 13-16 and will be described below. 

At step 224, in addition to storing the signature for the 
current broadcast segment, the system also stores the length 
of the commercial advertisement and other pertinent infor- 
mation such as the latest date and time of day (date/time) of 
the broadcast of the segment. The length is used to determine 
how long to mute the output signal if the same commercial 
advertisement is again encountered. The latest date/time of 
broadcast can be employed to selectively erase signatures 
from the database based upon how recently the signatures 
had been encountered. More information regarding the con- 
tent and purpose of information stored in the database along 
with the signature will be described in greater detail below. 
After the database is updated at step 224, step 225 is 
performed to determine whether the detected commercial 
advertisement is the first of a group and if so continuous 
group-based muting, described above, is performed until at 
least the end of the minimum pre -determined group time 
period, assuming no user over-ride. Then, step 222 is 
executed to end the current broadcast segment evaluation 
process. 

Hence steps beginning at step 218 are performed only if 
the signature of the current broadcast segment does not 
match any in the database as determined at step 216. If there 
is a signature match at step 216 indicating that the current 
broadcast segment is a commercial advertisement, the sys- 
tem first determines whether the commercial advertisement 
is the first of a group, step 226, and then begins muting the 
output television signal, step 227, by using a mute control 
signal in the manner described above with reference to 
FIGS. 1 and 3. Accordingly, television signals output at step 
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208 are muted and will remain muted until the mute control 
signal is deactivated. If the advertisement is the first of a 
group, the muting will ultimately continue until the end of 
the predetermined minimum group time period as described 
above, assuming no user over-ride. If not, muting will 5 
continue until the end of the advertisement, again assuming 
no over-ride. The length of the commercial advertisement is 
then read from the database, step 228, such that system can 
determine the end of the current commercial advertisement 
segment. Hence, if a previously triggered group-based mul- 10 
ing terminates during the current segment, the system will 
continue to mute until the end of the segment. At step 230, 
the system updates the parameters stored with the signature, 
such as the latest date and time of broadcast. Then, at step 
232, the muting is terminated either upon user over-ride or 15 
upon the latter of the completion of the current commercial 
advertisement segment and the completion of the current 
group (as determined by the minimum predetermined group 
time period) if group-based muting had been triggered. The 
current commercial advertisement segment, as with all 20 
segments, end with an event marker which is detected at step 
204 triggering another broadcast segment evaluation pro- 
cess. 

Hence, FIG. 8 illustrates that broadcast segment evalua- 
tion processes are triggered upon the detection of each event 25 
marker. As discussed above, each broadcast segment evalu- 
ation process may take up to 60 seconds to determine 
whether the broadcast segment beginning at the event 
marker triggering the process is a commercial advertisement 
or not. During that period of time a number of other event 30 
markers may be detected and a broadcast segment evalua- 
tion process triggered for each one. Hence, a number of 
evaluation processes may proceed simultaneously. The reso- 
lution of one process may moot or obviate some of the 
latter-triggered processes. For example, if a first triggered 35 
process determines that the corresponding broadcast seg- 
ment is a commercial advertisement, then any event marker 
detected before the end of the broadcast segment thereby 
occurred during the commercial advertisement and therefore 
do not represent the beginnings of unique commercial 40 
advertisements, and the broadcast segment evaluation pro- 
cesses triggered thereby may be terminated. In other words, 
if a commercial advertisement is detected because a pair of 
event markers are found to be exactly 60 seconds apart, any 
other event markers detected during that 60 second period 45 
thereby occurred during the 60 second commercial and 
therefore can be ignored. On the other hand, if after 60 
seconds a first triggered process determines that a first event 
marker at the beginning of that 60 second period is not the 
beginning of a commercial advertisement, then any event 50 
marker detected after that first event marker may itself 
represent the beginning of a commercial advertisement 
segment, and the broadcast segment evaluation processes 
triggered thereby may not be terminated until the situation is 
further resolved. Conversely, if a second triggered process 55 
determines that the corresponding broadcast segment is a 
commercial advertisement, then any event markers detected 
before the beginning of that broadcast segment occurred 
before the commercial advertisement began and therefore do 
not themselves represent the beginnings of unique commer- 60 
cial advertisements, and any broadcast segment evaluation 
processes triggered thereby may be terminated. For clarity in 
illustrating the basic process of the invention, the details of 
the logic by which the resolution of one broadcast segment 
evaluation process affects others are not set forth in FIG. 8 65 
but may be readily incorporated within a computer program 
or other logic device configured for implementing the 
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method of the invention in accordance with the teachings 
provided herein. 

With reference to FIGS. 9-16 details of exemplary imple- 
mentation of some of the steps of FIG. 8 will now be 
described. 

As noted, step 212 of FIG. 8 operates to read a signature 
corresponding to an event marker from the broadcast signal. 
Any of a wide variety of techniques for defining the signa- 
ture of a broadcast segment and for reading and recording 
the signature may be employed. FIG. 9 sets forth one 
exemplary technique for reading a signature corresponding 
to an event marker from a broadcast signal. At step 236, the 
system waits a predetermined number of frames from the 
completion of the event marker. The system may wait, for 
example, 10 frames or the equivalent period of time. At step 
238, the system then records the current frame, e.g., the 
system takes a snap shot of the tenth frame, hereinafter 
referred to as the test frame. An exemplary test frame 240 is 
illustrated in FIG. 10. At step 242, the system determines, for 
each of a preselected set of signature test lines of the test 
frame, the average luminance level of the lines. The signa- 
ture test lines may be, for example, the first 64 odd lines of 
the frame beginning at line 23 or perhaps the 5th, 10th, 15th, 
etc. lines. A few exemplary signature test lines are identified 
in FIG. 10. The average luminance level of each line may be 
determined, for example, by routing the luminance signals 
for those lines through circuitry (not shown) configured to 
generate a test voltage level representative of the average 
voltage level of the luminance signal. At step 244 a deter- 
mination is made, for each signature test line, as to whether 
the average luminance level for that line exceeds a pre- 
determined threshold. This determination may be made, for 
example, by defining the threshold level in terms of a 
threshold voltage level and comparing the test voltage level 
with the threshold voltage level. At step 246, the system 
assigns a bit value of 1 to each line exceeding the threshold 
luminance level and a bit value of 0 otherwise. Of course, 
opposite bit values could alternatively be employed. At step 
248, the bit values are concatenated together yielding a 
single binary bit string bash code signature with one bit per 
test line of the test frame. An exemplary bit string 250 is set 
forth in FIG. 11. Execution then returns to step 212 of FIG. 
8 via step 249 where the signature is compared against those 
in the database. 

Hence, a bit string signature is evaluated based upon the 
average luminance level of selected lines of one selected 
frame following the event marker. Any bit string of suitable 
length may be employed. If too few test signature lines are 
selected yielding a short bit string, the bit string may not 
have sufficient bits to uniquely identify the broadcast seg- 
ment. Accordingly, it is preferred that at least 32 signature 
test lines are selected yielding a bit string of at least 32 bits. 
With 32 bits, it is reasonably unlikely that test frames from 
different program segments will have the same bit string and 
hence, the bit string can be used to uniquely identify each 
unique broadcast segment. Also, a test frame occurring fairly 
promptly after the event marker should be selected. If a test 
frame is selected that occurs too late after the event marker, 
the system will not be able to promptly determine whether 
the signature of the test frame matches any of the recorded 
signatures and will therefore not be able to promptly begin 
real-time muting. Hence, it is preferred that the test frame 
occur no longer than about one second after the event 
marker. In alternative implementations, signature test lines 
may be drawn from two or more separate test frames. 

Thus FIGS. 9-11 set forth exemplary techniques for 
reading a signature wherein a bit string is generated based 
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upon whether average luminance levels of selected signature 
test lines of a selected test frame exceed a threshold level. Of 
course, the same test signature lines, the same test frame and 
same threshold level must be employed each time to ensure 
that the same bit string is generated every time the same 5 
broadcast segment is encountered. Hence, care should be 
taken in detecting and comparing signatures to compensate 
for skew or other factors in the television signal. Details of 
one possible system for detecting and comparing signatures 
which addresses skew and other factors is set forth in VS. 10 
Pat. No. 5,504,518 summarized above. 

If the television signal is digital, average luminance levels 
need not be employed. Rather, the signature is simply based 
on selected bits within one or more selected digital frames. 
Skew and other related problems are also avoided or more 15 
easily overcome. 

As noted, step 218 of FIG. 8 operates to determine if the 
current broadcast segment nevertheless corresponds to the 
beginning of a commercial advertisement even though the 
signature of the segment beginning with the event marker 20 
triggering the evaluation process does not match any in the 
database. FIG. 12 sets forth one exemplary technique for 
making the determination. An overview of the technique 
was provided above in connection with FIGS. 4-7. At step 
252 a set of predetermined time periods based on standard 25 
commercial advertisement lengths are read from a memory. 
Examples include 15 seconds, 30 seconds, 60 seconds. At 
step 254, the system sets appropriate times to begin waiting 
each of the pre-determined time periods. At step 256, the 
system determines whether a second event marker occurs at 30 
any of the pre-determined time periods following the first 
event marker. For example, the system first waits 15 seconds 
from the time of the first event marker and detects whether 
the television signal includes an event marker at that point 
in time. If not, the system waits until 30 seconds from the 35 
first event marker and so on. If a second event marker is 
detected at any of the predetermined times, step 258, then 
step 260 is performed wherein the broadcast segment begin- 
ning at the first event marker is identified as a commercial 
advertisement. At step 262, the system determines the length 40 
of the commercial advertisement by comparing the time the 
second event marker was detected with that of the first 
marker. Execution thereafter returns through step 264 to step 
218 of FIG. 8. If at step 258, no event marker is found at any 
of the pre-determined time periods following the first event 45 
marker, then step 266 is performed wherein the broadcast 
segment starting at the first event marker is identified as not 
being a commercial advertisement segment and again execu- 
tion returns to step 218 of FIG. 8. 

Step 224 of FIG. 8 operates to store the signature and so 
other pertinent information in the database if the broadcast 
segment is found to be a commercial advertisement. The 
database, however, may be full. A determination then needs 
to be made as to which previously stored signatures, if any, 
should be overwritten to make room for the newly recorded 55 
signature. FIGS. 13-16 set forth details of one exemplary 
technique. At step 268, the system determines whether the 
database is full and if not, step 270 is performed to store the 
signature, the length of the commercial advertisement for the 
segment corresponding to the signature and the date/time the 60 
event marker marking the beginning of the commercial 
advertisement was detected. Other information may also be 
recorded for the purpose of allowing the system to intelli- 
gently decide which records to erase as the database 
becomes full. For example, if the system is configured to 65 
detect commercial advertisements even while the television 
set is otherwise not in use (as will be described in greater 
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detail below with reference to FIG. 17), the system also 
stores an in -use weighting factor, which may be a single bit 
value, indicating whether the commercial advertisement was 
detected during a period of time when the television was not 
otherwise in use or whether it was detected while the 
television was in use. The system may also increment a 
counter identifying the number of times the commercial 
advertisement is encountered. 

FIG. 14 sets forth an exemplary database (for use as 
signature memory until 128 of FIG. 1) having, for each 
unique signature stored therein, entries for the first broadcast 
date/time, the latest broadcast date/time, the number of times 
the commercial advertisement is encountered and the in -use 
weighting factor. When the signature entry is initially 
recorded (step 224 of FIG. 8), the first date/time and the 
latest date/time are the same, the number of broadcast is one 
and the in -use weighting factor is either set to 1 or 0 
depending upon whether the television set is currently in 
use. 

Within FIG. 13, if the database is found to be full at step 
268 then step 272 is performed wherein the system selects 
one of the previously stored records in the database to be 
overwritten based upon, for example, the latest date/time 
that the previous signatures were encountered, the first 
date/time the signatures were encountered, etc. For example, 
depending upon system pre-programming, the system may 
overwrite whichever entry was least recently written to the 
database as indicated by the latest date/time of broadcast 
information. This technique has the advantage of overwrit- 
ing signatures corresponding to commercial advertisements 
not encountered recently and thereby keeping the database 
up to date. If such is the case, no further information need 
be recorded along with the signature other than the latest 
date/time which must be updated upon each detection of the 
signature (see step 230 of FIG. 8). As another example, the 
system may overwrite whichever signature record has the 
lowest frequency of occurrence. In such a case, the system 
calculates a frequency of occurrence based upon the first 
date/time of entry, the latest date/time of entry and the 
number of broadcasts encountered. This technique has the 
advantage of overwriting signatures not frequently encoun- 
tered even though the signature may have been recently 
encountered. 

As another example, the system may overwrite whichever 
signature simply has the least number of encountered broad- 
casts as represented by the number of broadcasts counter. To 
ensure that newly stored signatures are not immediately 
overwritten the system may also look at the earliest time of 
broadcast and overwrite a signature only if it was recorded 
some minimum amount of time earlier. 

If the system is configured to search for commercial 
advertisements even while the television set is not otherwise 
in use, the system may be configured to overwrite signatures 
encountered during times that the television was not other- 
wise in use first before overwriting signatures encountered 
while the television set was in use. TTiis has the advantage 
of eliminating signatures that correspond to commercial 
advertisements not ordinarily encountered while the viewer 
of the television set typically watches television. In this 
regard, whenever a signature is first encountered, the system 
simply determines whether the television set is on and if so, 
sets the in-use bit to 1 if the set is on and 0 otherwise. 
Thereafter, if the signature is encountered while the in-use 
bit is set to 0 and while the television set is on, the in-use bit 
is re-set to 1. Once the in-use bit is set to 1 it is not reset to 
0 until the record is erased. Hence, the in-use bit indicates 
whether the signature has been encountered at least once 
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while the television set was on. In any case, if a signature 
record needs to be overwritten, records having in- use bits set 
to 0 are overwritten first before any records having in -use 
bits is set to 1. 

In yet another example, the system may overwrite which- 5 
ever entry was first written to the database as indicated by 
the first date/time of broadcast information. If such is the 
case, the memory is preferably implemented, as illustrated in 
FIG. 15, as a FIFO 274 such that no explicit dateAime value 
need be stored. If implemented as set forth in FIG. 14, the 10 
memory is preferably a RAM. 

In yet another implementation, illustrated in FIG. 16, the 
database is capable of storing multiple records per signature. 
More specifically, database 276 of FIG. 16 stores a record 
identifying the broadcast dateAime of each detection of each j 5 
commercial advertisement signature. Accordingly the data- 
base maintains a record of when each commercial adver- 
tisement is broadcast. Exemplary signatures #1,-#1^ and 
#2 1 -#2 Af are shown. Periodically, the contents of the data- 
base are output to provide a list, perhaps for broadcast 20 
verification purposes, of the date and time of each broadcast 
of each commercial advertisement. Preferably, the database 
is of sufficient size such that records do not need to be 
overwritten prior to outputting the date/time broadcast infor- 
mation. 25 

As noted above, in one implementation the system is 
configured to operate in a background mode to detect 
commercial advertisements and record the corresponding 
signatures even while the television set is not otherwise in 
use. This is provided in part to allow the system to quickly 30 
build a database of commercial advertisements signatures 
such that, when the television set is in use, there is a higher 
probability that commercial advertisements encountered 
will be muted. The background mode is particularly advan- 
tageous for use when the system is initially activated with 35 
few, if any, pre-stored signatures. 

FIG. 17 sets for a method for implementing the back- 
ground mode. Initially, at step 300, commercial detecting 
and muting system 108 (FIG. 1) is powered-up with audio 
and video outputs to television monitor 104 (also FIG. 1) 40 
disabled. At step 302, the database is examined to determine 
whether it is complete. Depending upon p re-programming, 
the database may be regarded as being complete if it is full 
or perhaps if it holds some threshold number of signature 
records. Assuming that the database is complete, then step 45 
304 is performed where the system waits for user input to 
turn the television monitor on. Hence, no background pro- 
cessing is initiated. Once the monitor is activated, the audio 
and video are enabled, step 306, and the system begins 
receiving a broadcast signal, step 308. As with the basic 50 
method of FIG. 8, the system searches for event markers, 
step 310, and if event markers are detected, step 312, the 
system begins a broadcast segment evaluation process to 
determine whether the broadcast segment is a commercial 
advertisement, step 314. Details of the broadcast segment 55 
evaluation process are set forth in FIGS. 8-16 and have been 
described above. Then, regardless of whether an event 
marker has been detected or not, step 316 is performed to 
output the received broadcast signals to the monitor. Since 
audio and video had previously been enabled, the broadcast 60 
signal is thereby displayed by the television monitor, 
although perhaps muted if currently displaying a detected 
commercial advertisement. At step 318, the system deter- 
mines whether the user has de-activated the television 
monitor and, if not, step 308-318 are again performed. In 65 
this manner, steps 308-318 are repeated in a loop until the 
television monitor is turned off— at which time step 320 is 
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performed to again disable audio and video. At step 302, the 
database is again examined to determine if it is still complete 
and if so the system waits at step 304 until the user again 
activates the television monitor. Step 302 is performed 
following step 320 because, in some implementations, the 
user is provided with the capability of clearing all or part of 
the commercial advertisements signature database while 
operating the television set. Hence, the commercial adver- 
tisement database may no longer be complete. 

If at step 302, the database is not complete, either because 
the system is newly installed and has not yet built up a 
complete database or because the user has cleared the 
database, execution proceeds to step 322 where the system 
determines whether the user has activated the television 
monitor and if so execution proceeds to step 306 and 
following for non-background commercial advertisement 
detection processing as already described. If the user has not 
activated the television monitor, execution instead proceeds 
immediately to step 324 where background processing 
begins. (It should be noted that step 322 differs from step 
304 where the system waits for tie user to activate the 
television monitor. With step 322, the system does not wait 
for the user to activate the television monitor but immedi- 
ately triggers either background or non-background 
processing). 

Background processing begins at step 324 wherein broad- 
cast signals are received. As with the basic method of FIG. 
8, the system searches for event markers, step 326, and if 
event markers are detected, step 328, the system begins a 
broadcast segment evaluation process to determine whether 
the broadcast segment is a commercial advertisement, step 
330. Details of the broadcast segment evaluation process are 
set forth in FIG. 8-16 and have been described above. Then, 
regardless of whether an event marker has been detected or 
not, execution returns to step 302 to again evaluate whether 
the database is complete. It should be noted that in the 
background processing loop of steps 302-330, there is no 
step of outputting the broadcast signal. Since audio and 
video had previously been disabled, no output signal is 
needed. 

If the database remains incomplete and the television 
monitor remains deactivated, the background processing 
mode of steps 302-330 is performed in a loop to build a 
signature database. In this regard, every time a broadcast 
segment evaluation process is triggered at step 328 which 
detects a commercial advertisement, the signature for the 
advertisement is added to the database in the manner 
described above with reference to FIGS. 8-16. The back- 
ground processing loop ends either when the database 
becomes complete (at which time the system waits at step 
304 for user input to activate the television monitor) or when 
the user activates the television monitor at step 322 even 
though the database is not yet complete. In either case, 
non-background processing is then performed. 

Thus FIG. 17 sets forth a method for operating the system 
in a background mode to build a signature database even 
while the television set is not otherwise in use and further 
sets forth a technique for determining whether background 
or non-background processing should be performed based 
upon the completion state of the database and upon whether 
the user activates the television monitor. In other 
implementations, rather than detect whether the database is 
full and activate the background processing mode if it is not 
full and the user has not activated the monitor, the system 
merely operates in background mode for some predeter- 
mined period of time after the initial system power up 
(unless the user has activated the monitor). The pre- 
determined period of time may be, for example, two weeks. 
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As noted, a system employing the above-described tech- video of television signals while operating in television 

niques may be implemented either as a plug-in module or as mode in the same manner as described above. Hence, 

an external device. If a plug-in module, the module is commercial advertisements or other unwanted broadcast 

interconnected between an output of a tuner portion of the material are selectively muted. When implemented via 

television set and an input to an audio/video display portion 5 hardware, a plug-in module as described above is provided, 

of the television set to output selectively muted or otherwise The plug-in module either has its own memory for storing 

altered audio and video signals. If external, the system may signatures and other information or is configured to employ 

be interconnected to a television set as shown in FIG. 18. a disk drive or other storage unit 512 of the PCTV for such 

A selective muting unit 400 is connected to audio 402 and purposes. Signatures may be stored temporarily during 

video 404 output jacks of a television set 406. The audio and 10 processing with a RAM (not shown) of the PCTV to 

video output jacks are connected internally to a tuner portion expedite processing, if implemented in software, the soft- 

(not separately shown) of the television set Selective mut- ware may, for example, run on a microprocessor of the 

ing unit 400 receives the audio and video signals from the PCTV to process digitized portions of the television signal 

tuner via the audio and video output jacks and determines, (provided by an analog to digital converter not separately 

using the techniques described above, when to initiate and 15 shown) for muting purposes, again using a disk drive or 

terminate muting. An infrared muting transmitter 408 other storage unit 512 for storing signatures and the like. IF 

receives an electrical muting signal from muting unit 400 the PCTV is configured to receive digitized television 

and outputs an infrared signal to an infrared input 410 of the signals, then no tuner or analog to digital converter may be 

television set. The infrared muting signal is set to emulate needed. 

the infrared signal normally received by the television set ^ When in PC mode, the PCTV does not operate to perform 
from an infrared remote control unit (not separately shown) any muting, unless the PC simultaneously displays the 
when a user presses a mute button on the remote control. In television signals within a window of monitor 506. Prin- 
this manner, the audio is muted during detected commercial ciples of the invention may be additionally applied to detect 
advertisements or other unwanted broadcast material as and selectively process other types of incoming data signals 
described above. The mute signal is transmitted at the ^ besides television signals to mute or otherwise alter 
beginning of a detected commercial advertisement and a unwanted portions of the signals, including signals other- 
second un-mute signal is transmitted either at the end of the wise processed only in PC mode including, for example, 
commercial advertisement or at the end of a group of advertisement portions of input web pages, 
commercial advertisements. What has been described is a method and apparatus for 

If the television set is provided with a video mute capa- 30 detecting commercial advertisements by comparing signa- 

bility triggered by a video mute infrared signal from a tures of received broadcast segments with a database of 

remote control unit, then infrared muting transmitter 408 stored signatures corresponding to known commercial 

additionally transmits the video mute signal to thereby advertisements and for muting any segments having a sig- 

provide both video and audio muting. Other television nature matching any of the stored signatures in real-time, 

functions responsive to infrared input control may be addi- 35 For segments having signatures not found in the database, 

tionally or alternatively exploited. Also, if the television set the system nevertheless determines whether the segment is 

is provided with audio and video input jacks along with the a commercial advertisement and, if so, adds the signature to 

audio and video output jacks, muting unit 400 may be the database. This latter determination is based upon known 

configured to output muted audio and video signals into the characteristics of commercial advertisements. In the exem- 

audio and video input jacks of the television set. In such an 40 plary embodiment, the determination is based upon the 

embodiment, the television set is set to a mode for outputting assumption that commercial advertisements are always pre- 

signals received via the tuner of the television set through ceded and followed by black frame/low audio event markers 

the audio and video output jacks and for displaying signals and are always 15, 30 or 60 seconds in length. This appears 

received through the audio and video input jacks. Hence, to be a fairly universal rule at least with television signals 

selectively muted television signals are display in the same 45 broadcast in the United States. However, if employed in 

manner as if the muting unit were mounted internally within connection with a television broadcast system wherein com- 

the television set. mercial advertisements are typically of different lengths, the 

Thus far, the invention has been described with respect to system of the invention may be pre-programmed with those 
exemplary embodiments employing television sets. FIG. 19 different lengths. In other implementations, other appropri- 
iUustrates an embodiment employing a PCTV 500 config- 50 ate rules or characteristics of commercial advertisements or 
ured to operate either in a television mode or a personal other selected segments may be employed either addition- 
computer (PC) mode. Depending upon the mode, PCTV 500 ally or alternatively for distinguishing the selected segments 
receives television signals via a cable 502 and an internal from other broadcast segments. 

tuner (not separately shown) or receives and transmits In general, almost any set of rules or characteristics of 

modulated data signals over a telephone line 504, or other 55 broadcast signals that serve to distinguish commercial 

data line such as an ISDN line. Alternately, both television advertisements or other selected segments from program 

and modulated data input and output may be via a single line segments may be employed to determine whether or not a 

such as a cable line. In any case, while in television mode, broadcast segment, for which there is not signature match, is 

PCTV 500 displays television signals via a monitor 506 and nevertheless a commercial advertisement. The rules are 

plays audio signals via speakers 508. While in PC mode, the 60 merely programmed or otherwise incorporated within sys- 

PCTV displays graphic images generated by computer pro- tem 108 of FIG. 1 to detect commercial advertisements and 

grams such as web browsers, word processors, spread sheets an appropriate signature or other distinguishing feature is 

and the like via monitor 506, plays any corresponding audio stored for application against future received broadcast 

via speakers 508 and receives user input via an keyboard signals. Indeed, suitable rules may be applied to detect 

510- 65 almost any desired selected segment within a television 

PCTV 500 is provided with hardware or software or both broadcast signal and not necessarily just commercial adver- 

(not separately shown) for selectively muting the audio and tisements. Moreover, principles of the invention may be 
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applied to signals other than television signal such as to 
radio broadcast signals. 

Moreover, although described with respect to television 
sets of PCTV's any device or computer configured to 
receive television or other broadcast signals can be provided 
with appropriate hardware or software or both for detecting 
and selectively processing portions of the broadcast signals 
in the manner described above. 

Although described with reference to an exemplary sys- 
tem which operates to mute a television signal during 
commercial advertisements, almost any other desired action 
may alternatively be triggered. For example, the system may 
completely cut off the broadcast video and audio feed and 
replace the video feed with, for example, a screen presenting 
an appropriate logo representative of the commercial deter- 
mination system such as the tradename or trademark of the 
system, perhaps Sound (OFF™. As another example, the 
system may change the reception channel upon detection of 
a commercial advertisements then return to the same chan- 
nel upon completion of the advertisement or group of 
advertisements. Such requires a two- tuner system. Also, as 
described above, the system may merely record the date and 
time of each commercial advertisement for broadcast veri- 
fication purposes. As can be appreciated, a wide range of 
alternative implementations are available consistent with the 
general principles of the invention. 

The preceding description of exemplary embodiments is 
provided to enable any person skilled in the art to make or 
use the invention. It should be appreciated that not all 
components necessary for a complete implementation of a 
practical system are illustrated or described in detail. Rather, 
only those components necessary for a thorough understand- 
ing of the invention have been illustrated and described. 
Furthermore, it should be understood that the exemplary 
embodiments of the invention described herein are merely 
illustrative of general principles of the invention which can 35 
be applied to other applications, to achieve other ends, and 
to remedy other problems not specified herein. Accordingly, 
the scope of the invention should not be limited to the 
exemplary embodiments described herein. 

I claim: 

1. A method for identifying selected segment of a televi- 
sion signal comprising the steps of: 
detecting a first event marker; 

detecting a signature pattern associated with the first event 
marker; 

comparing the signature pattern to stored signature pat- 
terns; 

if the signature pattern matches any of the stored signature 
patterns, identifying a pre-determined portion of the 
television signal following the first event marker as a 50 
selected segment; 

if the signature pattern does not match any of the stored 
signature patterns, analyzing signals following the first 
event marker to determine whether the signature pat- 
tern nevertheless corresponds to a selected segment 55 
and, if so, identifying a pre-determined portion of the 
television signal following the first event marker as a 
selected segment and storing said signature pattern of 
the selected segment along with the stored signatures; 

discarding selected stored signatures; 

wherein the method is performed by a television set either 
while the television signal is displayed for view by the 
television set or while the television set is otherwise not 
in use; 



40 



45 



60 



in-use factor being representative of whether the 
selected segment was encountered at any time while the 
television set was in use; and 
wherein the step of discarding selected stored signatures 
includes the step of identifying which signatures to 
discard based in part upon the in-use factor. 

2. A method for identifying selected segments of a tele- 
vision signal comprising the steps of: 

detecting a first event marker; 

detecting a signature pattern associated with the first event 
marker; 

comparing the signature pattern to stored signature pat- 
terns; 

if the signature pattern matches any of the stored signature 
patterns, identifying a pre-determined portion of the 
television signal following the first event marker as a 
selected segment; 

if the signature pattern does not match any of the stored 
signature patterns, analyzing signals following the first 
event marker to determine whether the signature pat- 
tern nevertheless corresponds to a selected segment 
and, if so, identifying a pre-determined portion of the 
television signal following the first event marker as a 
selected segment and storing said signature pattern of 
the selected segment along with the stored signatures; 

discarding selected stored signatures; 

whenever a selected segment is identified, storing and 
updating an identification of the television signal when 
the segment was most recently detected along with the 
signature of the segment; and 

wherein the step of discarding selected stored signatures 
includes the step of discarding the signature segments 
least recently detected. 

3. A method of controlling a television receiver compris- 
ing: 

(a) displaying a television signal; 

(b) detecting a first segment of the television signal; 

(c) capturing a signature pattern associated with the first 
segment of the television signal; 

(d) comparing the captured signature pattern to stored 
signature patterns; 

(e) if the captured signature pattern matches one of the 
stored signature patterns, controlling the television 
receiver to alter the displayed television signal; 

(f) if the captured signature partem does not match any of 
the stores signature patterns, storing the captured sig- 
nature pattern in a temporary storage location; 

(g) adding the captured signature pattern to the stored 
signature patterns if a second segment of the television 
signal is detected within a predetermined window of 
time following detection of the first segment. 

4. The method of claim 3 wherein controlling the televi- 
sion receiver to alter the displayed television signal com- 
prising muting an audio portion of the television signal. 

5. The method of claim 4 wherein the audio portion of the 
television signal is muted for a length of time stored in 
association with said one of the stored signature patterns. 

6. The method of claim 5 further comprising generating 
an audible alert when the length of time during which the 
audio portion of the television signal is muted expires. 

7. The method of claim 3 wherein controlling the televi- 
sion receiver to alter the displayed television signal com- 
prises muting a video portion of the television signal. 

8. The method of claim 7 wherein the video portion of the 
television signal is muted for a length of time stored in 



wherein said step of storing said signature pattern of the 65 association with said one of the stored signature patterns, 
selected segment along with the stored signatures 

includes the step of storing an in-use factor with the * * * * * 
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Cross Reference to Related Application 

This application claims the benefit of 
5 United States Provisional Application No. 60/097,504, 
filed August 21, 1998. 

Background of the Invention 

This invention relates to program guide 
systems, and more particularly, to techniques for 

10 notifying a user of television programs. 

Cable, satellite, and broadcast television 
systems provide viewers with a large number of 
television channels. Viewers have traditionally 
consulted printed television program listings to 

15 determine which programs were scheduled to be broadcast, 
on a particular day and at a particular time. 

Recent technological advances have allowed 
more convenient and advanced program guide services to 
be developed.. For example, passive television-based 

20 program guides have been developed that allow 

television viewers to view television program listings 
directly on their television sets. In addition, 
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interactive television program guide services have been 
developed that allow a service provider to deliver 
television program listings data to a user's local set- 
top box. An interactive television program guide 
application in the set -top box allows the user to 
display television program listings on the user's 
television set. In known interactive program guide 
systems, the program listings data made available to a 
user are limited to a predetermined time frame of 
program listings. 

Users may be interested in programs other 
than the programs in the available time frame of 
program listings. Such so-called non-frame television 
programs are those television programs which are 
outside the current display time frame. In contrast, 
in- frame television programs are those television 
programs which are inside the current display time 
frame. The current display frame for program guides is 
typically about two weeks or less, which is limited by 
the size of the program guide database that is 
maintained by the program guide. The frame can also be 
limited by the bandwidth for the distribution of the 
data and limited by the ability to collect accurate 
data (e.g., collected listings which are farther out in 
time can be less accurate) . As an example of the 
category of non- frame television programs, consider 
movies presently showing in movie theaters but soon to 
be released on pay-per-view cable. Because cable 
customers are not provided with information that lets 
them know that the movie they are planing to see at the 
local movie theater will be available shortly on cable, 
cable companies stand to lose revenue to movie 
theaters. Similar problems exist for televised 
sporting events and the like. 



Thus, users are still faced with the problem 
of tracking television programs that they desire to 
watch but which have not yet been scheduled (i.e., non- 
frame television programs) . 

It is therefore an object of the present 
invention to provide a way to notify users of upcoming 
television programs. 

Summary of the Invention 

This and other objects of the invention are 
accomplished in accordance with the principles of the 
present invention by providing program guide 
notification methods and systems. 

Users may be interested in programs which are 
not in a current program listings time frame of a 
program guide. For such programs, the program guide 
can provide a notification option. The notification 
option can provide the user with an opportunity to 
request that a notification to be received when a 
program of interest is determined to be in the current 
time frame. Notifications can be message 
notifications, reminder notifications, e-mail 
notifications, etc. The user can be provided with the 
opportunity to setup the parameters for notifications. 
The program guide can provide the user with the 
opportunity to view a list of notifications. 
Notifications can be set by selecting an advertisement 
for an upcoming program that is not presently in the 
current time frame, by entering a title of a program, 
by selecting a program from a coming soon display 
screen, etc. A notification can include information on 
the program for which the notification is being 
provided and can include user-selectable options. The 
program guide can monitor the user's notification 



requests to collect information that can be used in 
marketing, scheduling programs, advertising, etc. 

According to one aspect of this invention, a 
method of providing interactive options for non- frame 
television programs on a display screen of a system 
(e.g., a television program guide system) is provided. 
The method includes: (1) presenting on the screen 
(e.g., a program guide screen) at least one option that 
corresponds to a non- frame television program, 

(2) allowing the user to select a first option, and 

(3) providing the user a service associated with the 
first option. 

In another embodiment, an interactive method 
of notifying a user of non- frame television programs 
that are now in-frame for use with a system (e.g., a 
television program guide system) is provided. This 
method includes: (1) receiving non- frame data that 
corresponds to at least one non- frame television 
program, and (2) when the non-frame program is in- 
frame, notifying a user that the program will be 
broadcast in the current time -frame by providing the 
user with at least one interactive option that 
corresponds to the data. 

In yet another embodiment according to this 
invention, a method for providing notifications of non- 
frame television programs to a user using a system 
(e.g., a television program guide system) is provided. 
The method includes: (1) allowing a user to specify a 
non-frame television program outside of the current 
display time frame, and (2) notifying the broadcast 
time to the user when the non- frame television program 
is now in the current time frame. In this embodiment, 
notification can be either non- interactive or 
interactive. In one non- interactive example, the 



method can involve sending a notification (or providing 
a notification order screen) with information regarding 
one or more selected non- frame television programs that 
are to be aired without a user-selectable option. In 
contrast, an interactive example might include a user- 
selectable option. 

A coming soon program display screen can be 
provided according to this invention. Included on the 
coming soon program display screen are non- frame 
television programs which are outside the current 
display time frame. 

If a user selects a program or a pay-per-view 
event that has not yet been included in the program 
listings of the current time frame, additional 
information related to the program or event can be 
displayed in a program information box. In addition to 
viewing this additional information on the program 
selected, the user can order a notification to notify 
the user when the selected program will be broadcast. 
Options can also be provided that allow the user to 
determine when notifications will be generated. 

If desired, a notification can be set by 
directly entering the title of a television program for 
which notifications are desired. 

As mentioned above, the user can also view a 
list of all currently requested notifications. If 
desired, information, such as the program name, type of 
notification, and the date and time the request was 
submitted can be displayed on the list for each 
notification. Entries can be added to the current 
notification list as soon as the user submits a new 
request. In addition, the user can cancel notification 
entries. 



In yet a further aspect of the invention, 
electronic mail notifications can be set and generated 
based on other preferences which the user can specify. 

Further features of the invention (including 
systems for implementing these methods) , its nature and 
various advantages will be more apparent from the 
accompanying drawings and the following detailed 
descriptions of the preferred embodiments. 

Brief Description of the Drawings 

FIG. 1 is a diagram of an illustrative 
television program guide system in accordance with the 
present invention. 

FIG. 2 is a diagram of an illustrative main 
program guide menu display screen in accordance with 
the present invention. 

FIG. 3 is a diagram of an illustrative 
current listings display screen from which the program 
listings in the current time frame can be available in 
accordance with the present invention. 

FIG. 4 is a diagram of an illustrative 
notifications display screen which can provide an 
illustrative notifications list in accordance with the 
present invention. 

FIG. 5 is a diagram of an illustrative 
messages display screen which can include a list of 
messages in accordance with the present invention. 

FIG. 6 is a diagram of an illustrative 
message display screen in accordance with the present 
invention. 

FIG. 7 is a diagram of an illustrative 
notification display screen in accordance with the 
present invention. 



FIG. 8 is a diagram of an illustrative 
reminder display screen in accordance with the present 
invention. 

FIG. 9 is a diagram of an illustrative set 
message display screen in accordance with the present 
invention. 

FIG. 10 is a diagram of an illustrative e- 
mail display screen in accordance with the present 
invention. 

FIG. 11 is a diagram of an illustrative new 
notification display screen in accordance with the 
present invention. 

FIG. 12 is a diagram of an illustrative 
reminder list display screen in accordance with the 
present invention. 

FIG. 13 is a diagram of an illustrative 
coming soon display screen in accordance with the 
present invention. 

FIG. 14 is a diagram of an illustrative 
coming soon display screen in accordance with the 
present invention. 

FIG. 15 is a diagram of an illustrative setup 
display screen in accordance with the present 
invention. 

FIG. 16 is a diagram of an illustrative 
notification setup display screen in accordance with 
the present invention. 

FIG. 17 is a flow chart of illustrative steps 
involved in providing a notification feature in 
accordance with the present invention. 

FIG. 18A is a diagram of an illustrative 
program guide screen containing television program 
guide listings in accordance with the present 
invention. 



FIG. 18B is a diagram of an illustrative 
coming soon program screen in accordance with the 
present invention. 

FIG. 19 is a diagram of an illustrative 
notification screen that allows a user to enter 
information for ordering notifications in accordance 
with the present invention. 

FIG. 2 0 is a diagram of an illustrative 
notification screen that allows the user to enter more 
specific information for ordering notifications in 
accordance with the present invention. 

FIG. 21 is a diagram of an illustrative 
notification in accordance with the present invention. 

FIG. 22 is a diagram of an illustrative 
notification screen listing all current notification 
orders in accordance with the present invention. 

FIG. 23 is a diagram of an illustrative 
notification screen that allows a user to enter a 
program title and to select among various options when 
ordering a notification in accordance with the present 
invention. 

Detailed Description of the Preferred Embodiments 

An illustrative interactive television 
program guide system 10 in accordance with the present 
invention is shown in FIG. 1. Main facility 12 can 
contain a back office processor 14 for organizing data, 
such as television program guide listings data, as will 
be described in greater detail below. Main facility 12 
can also contain a program guide database 16 for 
storing program guide information, pay-per-view 
ordering information, television program promotional 
information, etc. Some of the information stored in 
database 16 can be from an external data source 18. 



Back office processor 14 is used to process the 
information stored in database 16. 

Information from database 16 can be 
transmitted to multiple television distribution 
facilities, such as television distribution facility 20 
via communications links, such as communications 
link 21. Only one television distribution facility is 
shown in FIG. 1 to avoid over-complicating the 
drawings. Each Link 21 can be a satellite link, a 
telephone network link, a cable or fiber optic link, a 
microwave link, a combination of such links, or any 
other suitable communications path. If it is desired 
to transmit video signals over link 21 in addition to 
data signals, a relatively high bandwidth link, such as 
a satellite link is generally preferable to a 
relatively low bandwidth link, such as a telephone 
line. 

Television distribution facility 20 is a 
facility for distributing television signals to users. 
Television distribution facility 20 can be, for 
example, a cable system headend, a broadcast 
distribution facility, a satellite television 
distribution facility, or any other suitable 
distribution facility. 

The program guide information transmitted by 
main facility 12 to television distribution facility 20 
includes television program listings data for current 
and future television programs. The television program 
listings data for each program preferably includes 
numerous program characteristics, e.g., the title of 
the program, the channel for the program, a scheduled 
broadcast time (start time) and an ending time (or 
duration) . Other typical program characteristics 
include ratings, critics ratings, descriptions, 



categories (sports, movies, comedy, children, etc.), 
actors, etc. Transmitted program information can also 
include advertising information and pay program data, 
such as pricing information for individual programs and 
subscription channels, time windows for ordering 
programs and channels, telephone numbers for placing 
orders that cannot be impulse ordered, etc. 

Television distribution facility 20 
distributes television programming and program guide 
information to the user television equipment 26 or the 
user personal computer (PC) equipment 28 of multiple 
users via communications paths 30 and 32, respectively. 
PC equipment 28 is based on a central processing unit 
(CPU) 44 and can have a monitor 46 and keyboard 48. 
User television equipment 26 is based on a set-top 
box 34 and can have optional videocassette recorder 38, 
television 40, and remote control 42. 

Various techniques can be used to distribute 
television programming and program guide information. 
For example, television programming can be distributed 
over analog television channels and program guide data 
can be distributed over an out-of-band channel on 
paths 30. Data distribution can also involve using one 
or more digital channels on paths 30 and 32. Such 
digital channels can also be used for distributing 
television programming and other information. Multiple 
television and audio channels (analog, digital, or both 
analog and digital) can be provided to set -top boxes 34 
and PC equipment 28 via communications paths 30 and 32. 
In user television equipment 26, television programs 
can be tuned to by user's set-top box 34 and can be 
displayed by television 40. In user PC equipment 28, a 
video board or other such component can be used to tune 
to a television program, and the television program can 



be displayed on monitor 46. Program listings and other 
information can be distributed via communications 
paths 32 to PC equipment 28, which can have a cable 
modem or other such communications device for receiving 
or transmitting data. In addition, program listings 
and other data can be distributed by one or more 
distribution facilities that are similar to but 
separate from television distribution facility 20 using 
communications paths that are separate from 
communications paths 30 and 32. If desired, data from 
the main facility 12 can be distributed to PC 
equipment 28 over the Internet or other suitable 
communications path that does not involve the use of 
television distribution facility 20. In some 
configurations, program guide functions are provided in 
the form of an on-line program guide by using PC 
equipment 28 to access a web server, such as web 
server 29 over the Internet 31 or other such data 
network . 

The program guide system can use a client- 
server architecture whether or not the system involves 
use of the Internet. In a client-server arrangement, 
the program guide is partially implemented on a server 
and partially implemented on a client. The server can 
be a server, such as server 22 located at, for example, 
television distribution facility 20. Processors in the 
user equipment, such as set-top box 34 or CPU 44, can 
act as the client processors. Program guide data that 
are distributed to a client-server program guide at 
television distribution facility 2 0 can be stored in a 
database 24 at television distribution facility 20 that 
is maintained by server 22 . For clarity, the present 
invention is described primarily in the context of an 
interactive television program guide that is 
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implemented on user equipment using a set -top box 34 or 
CPU 44 that receives data from television distribution 
facility 20 , rather than in the context of program 
guides that are implemented using an on-line program 
guide configuration, are implemented partially on a 
server and partially on user equipment, or are 
implemented using any other such arrangements.- 

Certain functions, such as pay program 
purchasing can require set -top boxes 34 to transmit 
data to television distribution facility 20 over 
communications paths 30. If desired, such data can be 
transmitted over telephone lines or other separate 
communications paths. If functions, such as these are 
provided using facilities separate from television 
distribution facility 20, some of the communications 
involving set-top boxes 34 can be made directly with 
the separate facilities. 

User television equipment 26 has a receiver, 
which is typically a set-top box, such as set-top 
box 34, but which can be other suitable television 
equipment into which circuitry similar to set -top box 
circuitry has been integrated. For example, user 
television equipment 26 can be based on an advanced 
television receiver, such as a high definition 
television ( "HDTV 11 ) receiver. Program guide data can 
be distributed to set -top boxes 34 periodically or 
continuously and stored in database 36. In a client- 
server architecture, this database need not be 
maintained on the set -top box and may be distributed 
over a number of different locations. For example, the 
program guide can maintain a database, such as 
database 24, using a server, such as server 22 located 
at television distribution facility 2 0 or at some other 
appropriate location or locations. Television 
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distribution facility 20 can poll set-top boxes 34 
periodically for certain information (e.g., pay program 
account information or information regarding programs 
that have been purchased and viewed using local ly- 
5 generated authorization techniques) . 



processor to handle information distribution tasks. 
Each set -top box 34 preferably contains a processor to 
handle tasks associated with implementing a program 

10 guide application on the set-top box 34. User PC 

equipment contains CPU 44 to handle tasks associated 
with implementing a program guide application. 
Television distribution facility 2 0 can contain a 
processor for tasks associated with monitoring a user's 

15 interactions with the interactive program guides 

implemented on user PC equipment 2 8 and user television 
equipment 26 and for handling tasks associated with the 
distribution of program guide data and other 
information to user PC equipment and user television 

20 equipment 26. Moreover, the processor of television 
distribution facility 20 can be used to support the 
server functions of a client -server program guide. 



videocassette recorders 3 8 or other suitable recording 
25 devices allow selected television programs to be 

recorded. Each videocassette recorder 3 8 is connected 
to a respective television 40. To record a program, 
set-top box 34 tunes to a particular channel and sends 
control signals to videocassette recorder 38 (e.g., 
30 using an infrared transmitter) that direct 

videocassette recorder 38 to start and stop recording 
at the appropriate times. 



program guide implemented on set-top box 34, television 



Main facility 12 preferably contains a 



In user television equipment 26, 



During use of the interactive television 



program listings and other information can be displayed 
on television 40. Such program guide displays can be 
presented as an overlay on top of a television program 
to which the user has tuned with set -top box 34 or can 
be presented in place of such a program. Each set- top 
box 34, videocassette recorder 38, and television 4 0 
can be controlled by one or more remote controls 42 or 
any other suitable user input interface, such as a 
wireless keyboard, mouse, trackball, dedicated set of 
buttons, touch screen display remote, etc. Remote 
controls, such as remote control 42 , have various 
buttons that can be pressed by the user such as arrow 
keys (for directing on-screen movement of a highlight 
region, scrolling functions, etc.), an OK, select, 
enter, or other such selection key for making a 
selection (hereinafter referred to as an OK key) , 
channel number keys (for selecting a television 
channel) , a favorites key (to invoke functions related 
to user preferences) , a delete key (to delete menu 
items or to express a disinterest in certain user 
preferences) , etc. 

PC equipment, such as PC equipment 28, is 
based on a CPU 44 that maintains a database 45. A 
program guide implemented on PC equipment 28 can be 
used to access and display program guide information on 
monitor 46. User selections and inputs can be made by 
a keyboard 48, mouse, trackball, or other PC input 
devices. It is understood that the following 
discussion, although directed to user television 
equipment and set -top boxes, is applicable to program 
guides used in connection with PC equipment . 

Notification can be presented in a number of 
ways, such as "on-screen," "on-line," etc. As used 
herein, "on-line" notification includes notifications 



transmitted, at least in part, using the Internet. 
"On-screen" notification includes notifications 
transmitted to user- television equipment. "On-screen" 
and "on-line" notification methods are not mutually 
exclusive. For example, a notification can take the 
form of an electronic mail message "e-mail" that was 
transmitted via the Internet and displayed on user 
television equipment. 

A program guide display screen can include a 
notifications option. For example, as shown in FIG. 2, 
illustrative main program guide menu display screen 60 
can include notifications option 62 and six other 
options (current listings option 64, movies option 66, 
Pay-per-view option 68, messages option 70, coming soon 
option 72, and setup option 74). In addition, display 
screen 60 can include brand logo 82, system operator 
logo 84, current -time display portion 86, new message 
indicator 88, and advertisements 76, 78, and 80. 

Brand logo 82 can identify the brand of the 
interactive program guide (e.g., TV Guide). System 
operator logo 84 can identify the brand of the system 
operator. Current -time display portion 86 can contain 
the current time of day. New message indicator 88 can 
indicate that there are messages available. 

A program guide user can select an option by, 
for example, navigating a highlight region on a 
selectable option and pressing a data entry key, such 
as an OK key of a remote control. 

A notification option can be available for 
programs that are not in the current program listings 
time frame of the program guide. Such out -of -time- 
frame programs (i.e., non- frame programs) can include 
programs that are not in the current time frame of the 
program guide, but are in a schedule stored at a main 



facility. Such non- frame programs can further include 
unscheduled programs that have related information at 
the main facility indicating that they are coming soon 
(e.g., movies in current cinematic release, new series 
that have been announced but not yet scheduled, 
returning series which have not yet been scheduled, 
special events, etc.). Such non-frame programs can 
further include programs that the user can know of but 
for which the main facility has no available 
information. 

The program guide system can store a list of 
programs which are not in the current time frame but 
have been scheduled or which are upcoming programs 
that have not yet been scheduled. The list can be 
stored at a main facility, at a server at a television 
distribution facility, or at user television equipment. 
When a user is provided with the opportunity to enter a 
title for a program (see below) , the list can be 
searched for programs matching the entered title. The 
search can occur at the location where the list is 
stored. As programs move into the current program 
guide time frame, the comparison between the database 
programs in the current program guide time frame and 
programs in the user's notifications list can take 
place at any of those locations (e.g., main facility, 
server, or user television equipment) . 

For programs that are already in the current 
time frame of a program guide, a notification can not 
be available, although a reminder option can be 
available. 

When a user selects current listings 
option 64, the program guide can display a current 
listings display screen, such as illustrative current 
listings display screen 96 of FIG. 3. From the current 
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listings display screen 96, the program guide can 
provide the user with the opportunity to view a time 
versus channel listing of programs . The display 96 can 
provide the user with the opportunity to view and act 
on the program listings for the current time frame 
(e.g., one week, two weeks, etc.). The program guide 
can retrieve the program listings for the current time 
frame from a local database (e.g., database 24 or 36 of 
FIG. 1) . 

When a user selects movies option 66 or pay- 
per-view option 68, the program guide can display a 
listings display screens for the current time window of 
movies or pay-per-view programs. Each listings display 
screens can have a format which is similar to 
illustrative current listings display screen 96. It is 
to be understood that the format for display screen 96 
is illustrative and other formats for listings display 
screens can be employed. 

When a user selects a notification option, 
such as notifications option 62 of FIG. 2, the program 
guide can display illustrative notifications display 
screen 90 of FIG. 4. Display screen 90 can include 
list 92 of notifications that the user has previously 
set (e.g., notifications for the Blair Witch Project, 
Now and Again, and the Mummy) . The program guide can 
provide the user with the opportunity to select any one 
of the displayed notifications in order to delete or 
modify the selected notification. An add new 
notification option, such as option 94, can provide the 
user with the opportunity to add a new notification to 
the list. The add new notification option can be 
contained in list 92 or can be in another suitable 
position on display screen 90. 



When a user selects a messages option, such 
as messages option 70 of FIG. 2, the program guide can 
display a messages display screen, such as illustrative 
messages display screen 98 of FIG. 5. Display 
screen 98 can include a list of messages 100. The 
list 100 can include all messages that have been sent 
to this user's household. List 100 can include 
notification messages (e.g., the first two messages in 
list 100) . Other types of messages can be included in 
list 100, such as a message for informing the user of a 
cable outage. A user can select a message from 
list 100 by navigating a highlight region on an item in 
the list and pressing a data entry key, such as an OK 
key of a remote control . 

When a user selects a message from list 100, 
the program guide can display a message display screen, 
such as message display screen 102 of FIG. 6. When the 
selected message is a message other than a notification 
message, the program guide can display the message 
title, message text, and the option to keep or delete 
the selected message. When the selected message is a 
notification message, the program guide can display the 
program title as the message title and the program 
description as the message text. For a notification 
message, the program guide can display the scheduled 
time and channel for the program. The program guide 
can also display options which are related to the 
program. For example, the program guide can display 
order option 104, keep option 106, delete option 108, 
air times option 110, and reminder option 112. When 
the notification message is for a pay-per-view program, 
order option 104 can be displayed to provide the user 
with the opportunity to impulse order the program. 
When reminder option 112 is selected, the program guide 
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can provide the user with the opportunity to set a 
reminder for the program. When air times option 110 is 
selected, the program guide can provide the user with 
the opportunity to view a list of broadcast times. 
5 When keep option 106 is selected, the program guide can 
keep the message for future reference by the user, and 
when delete option 104 is selected, the program guide 
can delete the message. The program guide can 
determine which options can be displayed based on the 
10 program for which the notification message was 

received. For example, when the selected program is a 



W pay-per-view program, order option 108 can be 

,g displayed. Air times option 110 can be displayed when 

l/l there is more than one show time for the program, 

jtj 15 A message can have been sent to users to 

, } announce to users that a program is coming soon (e.g., 

|3 an e-mail advertisement for a future program which is 

j^y not in the current time frame) . For such messages, the 

j 3 program guide can display notify option 114. The time 

,q 20 and channel of the program can not be displayed in the 

message since the program is not yet listed in the 
current time frame of the program guide. When the 
notify option 114 is selected for a program that is not 
currently listed, the program guide can provide the 

2 5 user with the opportunity to set a notification for 

that program. 

For example, as shown in FIG. 7, when the 
user selects a notify option for a program that is not 
within the current time -frame of the program guide, the 

3 0 program guide can display a notification display 

screen, such as illustrative notification display 
screen 116 of FIG. 7. A notification display screen, 
such as illustrative notification display screen 116, 
can be displayed under certain other circumstances. 
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For example, when a user selects an advertisement for a 
program that is not listed in the current time frame, 
the program guide can display notification display 
screen 116. Also, when a user selects a program from a 
coming soon display screen (discussed below) , the 
program guide can display notification display 
screen 116. Moreover, when a user selects a program 
from a notifications display screen (e.g., 
notifications display screen 90 of FIG. 4), the program 
guide can display notification display screen 116 to 
provide the user with the opportunity to modify or 
delete that notification. 

Notification display screen 116 can include 
the title of the program and a description of the 
program. When a user has set a notification by 
entering a title of a program (discussed below) and a 
description is not yet available for that program, the 
program guide can simply display a description that is 
blank. Notification display screen 116 can include 
user-selectable options. For example, notification 
display screen 116 can include delete option 118, 
remind option 120, message option 122, and e-mail 
option 124. 

When delete option 118 is selected, the 
program guide can delete the notification. The types 
of notification desired can be specified with user- 
selectable options, such as remind option 120, message 
option 122, and e-mail option 124. The program guide 
can display a status indicator (e.g., a checkmark) in 
remind, message, and e-mail options 12 0, 122, and 124 
to indicate that the user has already selected a 
notification of this type for the program. 

When a user selects a remind option, such as 
remind option 120 of FIG. 7, the program guide can 
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display a reminder display screen, such as reminder 
display screen 126 of FIG. 8. Display screen 126 can 
provide the user with the opportunity to confirm the 
setting of a reminder for the upcoming program. When a 
remind option is selected for a program that a reminder 
notification has already been set, the program guide 
can provide the user with an opportunity to delete the 
reminder. 

When a user selects a message option, such as 
message option 122 of FIG. 7, the program guide can 
display a set message display screen, such as set 
message display screen 128 of FIG. 9. Display 
screen 128 can provide the user with the opportunity to 
confirm that a message should be sent to the user's 
household when the program is scheduled (e.g., when the 
upcoming program is now determined to be scheduled to 
be displayed during the current time frame) . When the 
message option (e.g., message option 122 of FIG. 7) is 
selected for a program for which the message option has 
already been set, the program guide can query the user 
to determine whether the message should be canceled. 

When a user selects an e-mail option, such as 
e-mail option 124 of FIG. 7, the program guide can 
display an e-mail display screen, such as e-mail 
display screen 130 of FIG. 10. E-mail display 
screen 130 can provide the user with the opportunity to 
select to have an e-mail notification sent to the 
user's e-mail address. E-mail display screen 130 can 
provide the user with the opportunity to enter an email 
address. Depending on the program guide system 
arrangement, an e-mail address can be entered with a 
remote control, a keyboard, a display remote, etc. If 
desired, a setup display screen, discussed below, can 
provide the user with the opportunity to enter an e- 
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mail address which would then be applied to all e-mail 
notifications. When an e-mail option (e.g., e-mail 
option 124 of FIG. 7) is selected and the user had 
already set an e-mail option earlier to receive an e- 
mail notification, the program guide can provide the 
user with the opportunity to delete the e-mail 
notification. 

As discussed above, when a user selects an 
advertisement for a non- frame program, the program 
guide can display a notification display screen, such 
as notification display screen 116 of FIG. 7. For 
example, a user can select interactive 

advertisements 76, 78, and 80 of FIG. 2, which are for 
programs that are not yet available in the current time 
frame of the program guide. As shown in FIG. 2, 
advertisement 76 can be for the Blair Witch Project, 
which is a future program that is coming to pay per 
view. Advertisement 78, an advertisement for Now and 
Again, can be for a new program to air in the fall 
television season which is now being advertised in the 
summer season. Advertisement 80 can be an 
advertisement for the upcoming new season of the E.R. 
television program which can commence in the next time 
frame. When a user selects any one of the three 
advertisements 76, 78, and 80, the program guide can 
display a notification display screen, such as 
notification display screen 116 of FIG. 7. 

When a user selects an add-new-notif ication 
option, such as add-new-notif ication option 94 of 
FIG. 4, the program guide can display a new 
notification display screen, such as new notification 
display screen 132 of FIG. 11. Display screen 132 can 
provide the user with the opportunity to enter the 
title of a program for which the user desires to 
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receive a notification. The title can be entered with 
a remote control, keyboard, a display remote, or 
another suitable data entry interface. When a title is 
entered, the title can be added to a notification list, 
such as notification list 92 of FIG. 4. The title can 
be added without a description. The program guide can 
determine whether the title for the program is in the 
current time frame, which can be stored in a local 
database of the program guide or at a remote server. 
When there is a match, the program guide can retrieve 
the matching information (e.g., program description). 
Display screen 132 can include options for providing 
the user with the opportunity to specify the types of 
notification desired (e.g., remind, message, e-mail, 
etc . ) . 

After a user has set reminders for upcoming 
programs, the program guide can display a reminder list 
display screen, such as reminder list display 
screen 134 of FIG. 12. The program guide can 
automatically display reminder list display screen 134 
shortly before the start of the upcoming program. 
Display screen 134 can include an overlay containing a 
list of reminders which have been set by the user. The 
overlay can be displayed on top of the video for the 
program that the user is currently watching. Display 
screen 134 can include a hide reminder option 136. 
When the user selects hide reminder option 136, the 
program guide can remove the overlay and return the 
user to viewing the current program. When the user 
selects a specific program in the list, the program 
guide can tune immediately to the channel showing that 
program. The program guide can provide the user with 
the opportunity to view the reminder list at any time 
until the listed programs end. The opportunity to 



selectively view the reminder list at any time can, for 
example, be provided by providing a predetermined key 
on a remote control. The list can include reminders 
other than notification reminders (e.g., reminders set 
for other features of the program guide) , which can 
have been configured with a different configuration 
setup than for notification reminders. 

Advance notification can be provided for 
programs that are coming soon. For example, the 
program guide can include a coming soon option, such as 
coming soon option 72 of FIG. 2. When coming soon 
option 72 is selected, the program guide can 
immediately display a list of upcoming programs (e.g., 
display screen 138 of FIG. 14) or can first display a 
selectable list of types of upcoming programs, such as 
coming soon display screen 140 of FIG. 13. Display 
screen 140 can contain a list of different types of 
upcoming programs in which the user can be interested. 
For example, as shown FIG. 13, the list can contain 
options for upcoming pay-per-view movies, upcoming 
special events, returning series, announced new series, 
upcoming programs on a specific channel, etc. 

When a user selects an option from display 
screen 140, such as pay-per-view movies option 142, the 
program guide can display a coming soon display screen, 
such as coming soon display screen 138 of FIG. 14. 
Display screen 138 can contain a list of programs that 
match the selected program type (e.g., pay-per-view 
movies) and that are not yet within the current time 
frame of the program guide. The list can be sorted 
alphabetically or by any other appropriate criteria. 
The list can include upcoming programs that have been 
scheduled to air outside the current time frame of the 
program guide. If desired, the list can include 
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upcoming programs that have not yet been scheduled. 
For example, the list can include upcoming programs 
that have been announced but not yet scheduled. The 
list can include an indicator for a program, such as 
indicator 144. Indicator 144 can indicate that the 
user has already set a notification for that program. 
When the user selects a program from the list, the 
program guide can display a notification display screen 
for that program, such as notification display 
screen 116 of FIG. 6. Notification display screen 116 
can provide the user with the opportunity to set, 
delete, or modify a notification for the program. 

The user can be provided with the opportunity 
to setup the configuration of notifications. For 
example, when a user selects a setup option, such as 
setup option 74 of FIG. 2, the program guide can 
display a setup display screen, such as setup display 
screen 14 6 of FIG. 15. Setup display screen 146 can 
include notifications setup option 148. When a user 
selects notifications setup option 148, the program 
guide can display a notification setup display screen, 
such as notification setup display screen 150 of 
FIG. 16. Display screen 150 can include reminder 
notice setup option 152, message notice setup 
option 154, and e-mail notice setup option 154. If 
desired, setup display screen 146 can include an option 
for specifying whether reminders will be displayed on a 
television screen, a display remote, or both. 

Reminder notice setup option 152 can provide 
the user with the opportunity to specify how far in 
advance of a start time of an upcoming program a 
reminder notification should be displayed. Values for 
how far in advance a reminder notice can be displayed 
can be set for example to be in the range of one to 
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f if teen minutes. If desired, reminder notice setup 
option 152 can not be provided and the program guide 
can use settings from a general reminders feature 
(e.g., non-notification reminders). When a program is 
not in a current time frame of the program guide, a 
reminder notification for that program can be set. 
Later, when the program guide determines that the 
program is now in the current time frame, the program 
guide can activate the reminder notification at an 
appropriate time before the start time of that program. 

When a user selects message notice setup 
option 154, the program guide can provide the user with 
the opportunity to specify the message notice time. 
The user can set the notice to receive a message 
notification for a program a few hours before the 
program or as far ahead of the program as the available 
program guide data permits (e.g., the length of the. 
current time frame) . If desired, the program guide can 
use a default value for the message notice time (e.g., 
one day) or can generate the notification message as 
soon as the program is added to the local program 
listings database of the program guide. 

When a user selects e-mail notice setup 
option 156, the program guide can provide the user with 
the opportunity to specify the advance notice time for 
e-mail notifications. The values or defaults for 
advance notice time for e-mail notifications can be the 
same as the values for message notifications. 

Notification setup display screen 150 can 
provide a user with the opportunity to specify an e- 
mail notification address. If desired, the program 
guide can allow the user to specify an individual e- 
mail address for each notification. If desired, the 
program guide can have ari e-mail address for the 
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household as part of an e-mail feature, and can use 
that address, rather than requiring the user to enter 
the address again. 



5 user with the opportunity to setup the configuration 
notifications for each individual program. For 
example, the program guide can display in a 
notification display screen (e.g., notification display 
screen 116 of FIG. 7) a notification setup option, such 

10 as notification setup option 119 of FIG. 7. When a 

user selects notification setup option 119, the program 
guide can display a notification setup display screen, 
such as notification setup display screen 150 of 
FIG. 16, to setup the notifications for that program. 

15 An individual program notification setup can be 
available in addition to a general setup for 
notifications. If desired, a setup for notifications 
can include a general option that specifies whether 
reminders, messages, or e-mails should be generated 

20 whenever a program with a notification comes into the 
current time-frame of the program guide. 



program, the program notification can be stored, 
preferably in non-volatile memory, at the user 

25 television equipment (e.g., user television 

equipment 44 of FIG. 1) . If desired, when the program 
guide system is in a client -server arrangement, the 
program notification can be stored at a server (e.g., 
server 22 at television distribution facility 20 of 

30 FIG. 1) . 



program guide (either at server 22 or user television 
equipment 26) , the incoming schedule items can be 
compared with the notification list (e.g., the programs 



If desired, the program guide can provide the 



When the user sets a notification for a 



When new schedule items are received by the 



listed in notification list 92 of PIG. 4) . When there 
is a match , the program guide can generate a 
notification for that program (e.g., reminder 
notification, e-mail notification, message notification 
etc.) . The type of notification desired can have been 
set earlier from a display screen, such as notification 
display screen 116 of FIG. 7. The advance notice time 
for the notification can have been set earlier from a 
display screen, such as notification setup display 
screen 150 of FIG. 16. 

For example, when a reminder notification has 
been set for a program that is determined to be in the 
current time frame, the program guide can schedule a 
reminder overlay for a specified number of minutes 
prior to the start of the program. When a message or 
e-mail notification has been set, the program guide can 
immediately generate a notification or can schedule a 
task to generate a message or e-mail notification an 
appropriate number of days or hours prior to the 
scheduled start of the program. 

Illustrative steps involved in providing a 
notification feature are shown in FIG. 17. At 
step 152, the program guide can provide the user with 
the opportunity to select or specify an upcoming 
program which is not yet in the current program 
listings time frame of the program guide. 

If desired, step 152 can include informing 
the user of an upcoming program (step 154) . For 
example, the program guide can display a coming soon 
display screen (e.g., coming soon display screen 140 of 
FIG. 13) . The program guide can display a different 
coming soon display screens based on category, type, 
actor, genre, etc. (e.g., coming soon display 
screen 138 of FIG. 14) . The program guide can send a 
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message to a user announcing an upcoming program that 
is not yet in the program listings database of the 
program guide. The program guide can display 
advertisements for upcoming programs which are not in 
the current program listings time frame. 

At step 152, the program guide can provide 
the user with the opportunity to select an upcoming 
program from a coming soon display screen (e.g., coming 
soon display screen 138 of FIG. 14), a message display 
screen (e.g., message display screen 102 of FIG. 6), a 
new notification display screen (e.g., by entering a 
program title or partial title from new notification 
display screen 132 of FIG. 11), a coming soon display 
screen for a category, such as genre, actor, rating, 
etc. (e.g., coming soon display screen 138 of FIG. 14 
for upcoming pay-per-view programs), etc. 

At step 156, the program guide can display 
information and user-selectable options that are 
related to the selected program. For example, the 
program guide can display a description, title, 
graphics, etc. about the selected program. As for 
user-selectable options, the program guide can provide 
the user with the opportunity to select an option to 
set a message notification (e.g., message option 122 of 
FIG. 7), to select an option to set an e-mail 
notification (e.g., e-mail option 124 of FIG. 7), to 
select an option to track a program (e.g., the program 
will be added to the notificat ion list but no 
notification will be generated) , to select an option to 
set a reminder notification (e.g., remind option 120 of 
FIG. 7), to select an option to purchase the selected 
program (e.g., order option 108 of FIG. 6), to select 
an option to have the program guide automatically tune 
to the channel carrying the select program at or about 
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the time of airing, to select an option to have the 
program guide automatically cause the program to be 
recorded when the program airs, etc. 

When the user selects an option, the program 
guide can provide the user with the opportunity to 
setup features related to the option. For example, the 
program guide can provide the user with the opportunity 
to select how far in advance a message, e-mail, or 
reminder notification should be sent. For purchase, 
tune, and record options, the program guide can provide 
the user with the opportunity to specify parameters, 
such as day of the week, time of day, etc. For 
example, the user can be allowed to select to a time 
range in which the program guide is authorized to 
purchase, tune, or record a program. 

At step 158, when the user has selected an 
option, the program guide can collect information on 
which programs and which options have been selected 
(step 158a) . The information can be collected at a 
central location (e.g., television distribution 
facility 20 of FIG. 1) . The information can be used to 
influence the scheduling of programming, the marketing 
of programming, the distribution of programming, and 
the targeting of advertisements, and to generate e-mail 
notifications . 

Step 158 can include providing the user with 
the opportunity to view or modify notifications set for 
programs (step 158b) . For example, the program guide 
can provide the user with the opportunity at any time 
to view a notifications list (e.g., notifications 
list 92 of FIG. 4) . An item in the notification list 
can be automatically removed from the list after a 
selected notification has been provided or can be 
removed a predetermined number of days after the 
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program has been received within the local program 
guide database (e.g., the current program listing time 
frame) . If desired, an item in the list can be removed 
after the program has been viewed or after the program 
is no longer available in the current time frame. The 
list can include an indication of which programs are 
presently in the current time frame of the program 
guide. From the notification list, the user can be 
provided with the opportunity to access a notification 
display screen (e.g., notification display screen 116 
of FIG. 7) in order to view, modify, or delete a 
notification that has been set for a program. 

Step 158 can include providing a notification 
for a selected program which is now in the current 
program listings time frame (step 158c) . When the user 
has set a notification message or reminder, the program 
guide can display a reminder notification or send a 
message notification when the program is determined to 
be in the current time frame of the program guide. If 
desired, a message notification can be sent or a 
reminder notification can be displayed at a 
predetermined notice time prior to the airing of the 
program. When the user has set an e-mail notification, 
an e-mail notification can be sent when it is 
determined that the program is in the current time 
frame of the program guide. If desired, the e-mail 
notification can be sent at a predetermined time prior 
to the airing of the program. The e-mail notification 
can be sent from a central facility (e.g., television 
distribution facility 2 0 of FIG. 1) or the program 
guide can generate the e-mail notification locally at 
user television equipment (e.g., user television 
equipment 26 of FIG. 1) . After a notification has been 
provided, the status of a notification in the 
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notification list can be modified (e.g., the program 
can be removed from the list when all the notifications 
which had been set by user have been performed) . At 
step 158c, the program guide can provide notification 
options, such as to tune or record the selected 
program. 

In the following discussion below with 
reference to FIG. * s 18 through 32, the present 
invention is discussed sometimes primarily in the 
context of a program guide system with a mouse, a 
keyboard, or both as a user interface device (e.g., an 
on-line PC based program guide system) . It is to be 
understood that the present invention is not limited to 
such an arrangement . 

FIG. 18A shows an illustrative program guide 
display screen that can be provided in a PC based 
program guide system. Display screen 218 contains 
program listings 220 that can be organized in channel 
order from top to bottom and by broadcast time from 
left to right. Cursors 222 and 224 can be used to 
navigate to earlier or later time periods, 
respectively. Browser cursors 226 and 228 allow a user 
to scroll through program listings 220. The user can 
also navigate through program listings 220 with time 
navigation buttons 230 to view program listings for 
different times in the day. Calendar buttons 232 can 
be used to view program listings for different days of 
a month. The user can choose between various available 
view options by selecting a desired time, channel, 
category, or search button from among view buttons 234. 

FIG. 18B shows an illustrative coming soon 
program display screen 310 that can be provided as an 
option from the display screen 218 of FIG. 18A, or can 
be accessed independently. The user can activate the 
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coming soon program display screen from display 
screen 218 of FIG. 18A by means of box 236. Included 
on the coming soon program screen are non-frame 
television programs which are outside the current 
display time frame. For example, in FIG. 18B, there 
are several movies, including Titanic, The General, and 
Saving Private Ryan, all of which (at least for the 
purposes of this description) have not yet been 
released on cable television. 

Program guide screen 310 can also include one 
or more markers 312 that indicate that information 
regarding other programs is now available to the user 
(e.g., a listing for Rushmore is in the current time 
frame) . When data regarding a non- frame television 
program is added, the non- frame television program by 
definition becomes an in- frame television program. The 
marker can be, for example, an icon, a color, a display 
order, and any combination thereof. More than one type 
of marker can be used on a single screen to indicate 
more than one type of information regarding the 
television program. For example, a star placed next to 
the listing can indicate that the program will be 
broadcast on a pay-per-view channel and an underline 
can indicate optional on-line links to more information 
and promotional offers. 

If a program is selected from coming soon 
display screen 310 of FIG. 18B, the user can be 
presented with a notification order display screen, 
such as notification order display screen 410 of 
FIG. 19. Notification order display screen 410 can 
contain user-selectable options for ordering a 
notification. Notifications can notify the user when a 
particular television program is to be broadcast. As 
defined herein, the term "broadcast" refers to the 
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process of airing scheduled television programs by 
traditional television broadcast techniques, cable 
systems, or satellite systems. It will be appreciated 
that the methods and apparatus described herein can 
5 also be used in connection with non- frame videos -on- 
demand . 



notification order display screen 410 , which allow the 
user to enter a name and electronic mail address, 

10 respectively, of the person receiving the notification. 
If desired, the user's name and electronic mail address 
can be automatically entered in fields 414 and 418 
based on information previously provided to the system 
(e.g., information provided when the user registered 

15 with the television program guide service) . Use of 

electronic mail to remind users of program events which 
are in the current time frame is known (See, United 
States Patent Application No. 08/987,740, filed 
December 9, 1997, which is hereby incorporated by 

20 reference in its entirety) . 



a computer located at the user's office, the user can 
wish to have notifications addressed to his home (i.e., 
using his personal Internet access account and the 

25 electronic mail address for the system at home) . 

Alternatively, the user can wish to have notifications 
sent to an office electronic mail address. If the user 
has a common electronic mail address for both home and 
the office, notifications can be accessed at either 

30 location. If desired, the system can provide more than 
one field 418 (e.g., field 421) so that notifications 
can be sent to more than one electronic mail address . 



FIG. 19 shows fields 414 and 418 in 



If the user is browsing program listings from 



Notification order display screen 410 of 
FIG. 19 can also provide several user-selectable 
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options that specify when and how often the user will 
be notified of the airing of selected non- frame 
television programs. For example, notification order 
screen 410 can contain notify me box 424. Selecting an 
option in notify me box 424 allows the user to specify 
how often the system will generate and send a 
notification to the user. As shown in FIG. 19 , the 
options in notify me box 424 can allow the user to be 
notified of the selected program only once, each time 
the program is being broadcast, each time the program 
is broadcast in a month, or at some other specified 
time. If "other" in notify me box 424 is selected, the 
user can be presented with screen 510 shown in FIG. 20. 
Such options can be selected with a mouse, keyboard, or 
any other suitable interface device. Screen 510 allows 
a user to specify a time period during which 
notifications will be generated and sent. The user can 
specify a time period by entering information in 
box 513 . 

Notification order display screen 410 of 
FIG. 19 can also contain how soon box 430. Selecting 
an option in box 430 allows the user to indicate how 
soon before the broadcast of the selected program 
notifications are to be generated and sent to the user. 
As shown in FIG. 19, box 430 can contain options that 
allow the user to be notified 1 hour, 1 day, 2 days, or 
another predetermined period of time before the 
selected program is broadcast. If "other" is selected 
in box 430, the user can be presented with display 
screen 510 of FIG. 21, which allows the user to specify 
a desired lead time before a scheduled event by 
completing box 515. 

Upon completing display screen 510 of 
FIG. 21, the user can submit the information that has 
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been entered by selecting enter button 517. The user 
can exit display screen 510 without submitting the 
information by selecting exit button 519. If the user 
selects either enter button 517 or exit button 519, the 
5 user can be returned to the previous screen, such as 
notification order display screen 410. 



boxes 424 and 430 of order display screen 410 
determine , respectively, how often and when 

10 notifications will be sent. For example, a user can 
select a program from coming soon program display 
screen 310 of FIG. 18B. When the user selects a 
program from display screen 310 of FIG. 18B, the user 
can be presented with order display screen 410 of 

15 FIG. 19. If the user selects the n l hour" option in 
box 430 and the "each time being broadcast" option in 
notify me box 424, the user will receive notifications 
(assuming fields 414 and 418 are completed with the 
user's own information) 1 hour before each broadcast of 

20 the program. 



notification order display screen 410 (FIG. 19) , the 
user can cancel the notification order by selecting 
. cancel box 480. Selecting cancel box 480 can return 
25 the user to display screen 310 of FIG. 18B. Upon 

completing order notification display screen 410, the 
user can submit a notification order for processing by 
selecting submit box 440. 



30 notification 600. Notification 600 can display 

electronic mail address 611 of the person to whom the 
mail is being sent, name 613 of the program for which 
the notification is being sent, and information on when 
the television program is to be broadcast. The user 



The options that the user selects in 



At any time during the completion of 



FIG. 21 shows an illustrative electronic mail 
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can delete notification 600 by selecting delete 
button 618. An advertisement 620 (text, graphics, 
video etc.) can be included in or attached to the 
notification if desired. The advertisement can be 
provided using information stored in a remote or local 
database, such as databases 16, 24, or 36. 

Notification 600 can also include interactive 
links to various screens and services, including, but 
not limited to a link to program guide display- 
screen 218 and a link promotional offers that can or 
can not be interactive. When a user selects interactive 
options 625 and 626, program events can be 
automatically ordered/ tuned, or recorded. If desired, 
these options can be automatically removed from the 
notification after being selected by a user to reduce 
clutter on the notification. The options can also be 
automatically removed after a predetermined period of 
time following a particular event. A particular event 
can occur (1) when data regarding the program is 
received by a local database, (2) when data regarding 
the program is received by a remote server, or (3) when 
the user is notified. 

Another aspect of the invention relates to 
management of one's notifications. If desired, 
notification order display screen 410 of FIG. 19 can 
contain view current box 483. If view current box 483 
is selected, the user can be presented with 
illustrative current notifications display screen 710 
of FIG. 22. Another way that the user can reach 
current notifications screen 710 of FIG. 22 is by 
selecting view current notifications button 23 3 in 
display screen 218 of FIG. 18A. Current notifications 
display screen 710 contains a list of all the user's 
currently requested notifications. Information, such 
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as the program name, type of notification, and the date 
and time submitted can be displayed. 



current notification entry for the program "Primal 
5 Fear". This notification was submitted by the user on 
November 1, 1997 at 3:03 p.m. and is set to notify the 
user one hour before each broadcast. Entries can be 
added to the current notification list as soon as the 
user sets a new notification (e.g., by selecting submit 

10 button 440 from order notification screen 410 of 
PIG. 3). When an item in list is selected (e.g., 
positioning a highlight region on an item and pressing 
a data entry key) , display screen 410 of FIG. 19 can be 
displayed to allow the user to modify the notification 

15 configuration for the selected item in the list. If 
desired, the user can provided with the opportunity to 
change the configuration for an item in the list from 
the same display screen, display screen 710. An item 
can be removed from the list (e.g., by positioning a 

20 highlight region on an item and selecting cancel 712) . 



new notifications box 231. New notifications box 231, 
which can be adjacent to the program navigation 
controls of display 218, allows the user to order 
25 notifications without using program listings 220. If 
new notifications box 231 is selected, the user can be 
presented with illustrative new notifications display 
screen 810 of FIG. 24. 



30 selectable options similar to the options contained in 
notification order display screen 410 of FIG. 19. For 
example, new notifications display screen 810 contains 
user information box 813, how soon box 830, and notify 
me box 824. However, new notifications screen 810 



FIG. 22, shows an illustrative example of a 



Another component of display screen 218 is 



New notifications screen 810 contains user- 
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allows the user to enter the title of a non-frame 
television program. 

If the user does not enter the exact title of 
the non- frame television program in new notifications 
display screen 810, the data in the database 16, 24, 
or 36 or any other suitable set of television program 
listings data can be scanned to find the program or 
programs that most closely match the program title 
indicated by the user. If several matches are found, a 
list of the program matches can be presented to the 
user and the user can choose the actual program 
desired. Once the user has chosen a program from the 
list, the program title in box 818 can be automatically 
updated. If no matches are found, or if the user does 
not accept one of the offered choices, the title can be 
left as entered. 

A user can exit new notifications display 
screen 810 at any time by selecting exit button 880. 
Selecting exit button 880 can return the user to 
display screen 218 of FIG. 18A. 

Upon completing new notifications display 
screen 810, the user can submit the notification order 
for processing by selecting submit box 840. Selecting 
submit box 84 0 allows program guide system 10 (FIG. 1) 
to process the request. The request can be processed 
in the same way a notification request can be processed 
when ordered from notification order display screen 410 
of FIG. 19. 

The user can reach current notifications 
display screen 710 of FIG. 22 from new notifications 
display screen 810 of FIG. 23 by selecting view current 
notifications button 883. 

The foregoing is merely illustrative of the 
principles of this invention and various modifications 
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can be made by those skilled in the art without 
departing from the scope and spirit of the invention. 
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